README.md

fhir-to-omop

This repository is dependent on the OHDSI Techstack from Gruhl which can be accessed here: https://gitlab.ukdd.de/pub/ohdsi/techstack

The current release of FHIR-to-OMOP is only compatible with OHDSI Techstack version v2024.01 and higher. All available versions of the OHDSI Techstack can be found here: https://gitlab.ukdd.de/pub/ohdsi/techstack/container_registry/343. Further releases of FHIR-to-OMOP are only compatible with the OHDSI Techstack version v3.5.1.

FHIR-TO-OMOP WIKI

The MII test data is available in deploy/init-test-data/MII_Test_data.zip in JSON format

Current Version v2.16.3

Run it

Hardware Requirements

• RAM: 24 GB
• CPU: 12 vCPU
• HDD: 1 TB

Start the ETL process with docker-compose:

cd deploy
cp sample.env .env
# change the value of parameters
docker-compose up

change the content in .env

General information

1. This ETL job can read FHIR resources from a FHIR Server (Blaze or HAPI)
2. The using of DATA_BEGINDATE and DATA_ENDDATE on a FHIR Server is not yet possible -> Problem with Blaze FHIR Server
3. The begin and end date are corresponding to the column last_updated_at in FHIR-Gateway.
4. If DATA_BEGINDATE and DATA_ENDDATE are set with default date, the filter on date in FHIR-Gateway is deactivated.
5. If DATA_BEGINDATE=2021-01-01 and DATA_ENDDATE=2021-01-01, the job will import all the data between 2021-01-01 00:00:00 and 2021-01-01 23:59:59 into OMOP.
6. The initial load of OMOP CDM runs as Bulk load (Recommended). At the beginning of the job, all tables, which are referenced to PERSON table will be emptied. The same applies to the tables in cds_etl_helper schema.
7. If FHIR-Gateway is reloaded with only new data, please run the Job as incremental load after the initial load, so that the tables in OMOP CDM will not be emptied at the beginning of the job.
8. If you want to load MedicationStatement resources to OMOP CDM in addition to MedicationAdministration, please set APP_WRITEMEDICATIONSTATEMENT_ENABLED to true. The default value is false.
9. For using Keycloak authentication please set the parameter APP_AUTHMODE to keycloak and set the lines starts with APP_KEYCLOAK with correct values. If you have questions or problems with Keycloak, please see Keycloak Documentation.

Connect to your FHIR-Server

To connect to your local FHIR-Server you probably have to add an adjusted cacerts to the running container. Therefore you have to add the certifacte of your root-ca to a cacerts file and place it in the deploy folder. The mount path is already created in the docker-compose file. If you don't need the cacerts simply comment out the line.

As BulkLoad

1. set the parameter APP_BULKLOAD_ENABLED to true.
2. set DATA_BEGINDATE and DATA_ENDDATE to import the desired date from FHIR to OMOP.

For bulkload there is the possibility to search for referenced data (persons, visit_occurrences or visit_details) in RAM or in the OMOP DB. The following parameter should be used for this purpose:

1. set the parameter APP_DICTIONARYLOADINRAM_ENABLED to true to search in RAM.
2. set the parameter APP_DICTIONARYLOADINRAM_ENABLED to false to search in OMOP DB.

For bulkload it is possible to run the job for different FHIR resources separately. All choosable steps are (case sensitive):

1. Observation
2. Procedure
3. Medication
4. MedicationAdministration
5. DepartmentCase
6. Condition
7. MedicationStatement
8. Immunization (GECCO dataset)
9. DiagnosticReport (GECCO dataset)
10. Consent (GECCO dataset)

As IncrementalLoad

1. set the parameter APP_BULKLOAD_ENABLED to false.
2. set DATA_BEGINDATE and DATA_ENDDATE to import the desired data from FHIR to OMOP.

If you want to test this ETL process locally, use the docker-compose.dev.yml to start the containers for OMOP and FHIR-Gateway pre-filled with sample data (see the init-test-data folder)

Configuration

sample.env

Variables	Default Values	Comments
BATCH_CHUNKSIZE	5000	The size of the batch
BATCH_PAGINGSIZE	200000	The size of reading size
BATCH_THROTTLELIMIT	4	The number of threads used for this job
LOGGING_LEVEL_ORG_MIRACUM	INFO	Change to 'DEBUG' for more information in logging
DATA_FHIRGATEWAY_JDBCURL	jdbc:postgresql://localhost:15432/fhir	The URL of FHIR gateway
DATA_FHIRGATEWAY_USERNAME	postgres	The user name of FHIR gateway
DATA_FHIRGATEWAY_PASSWORD	postgres	The password of FHIR gateway user
DATA_FHIRGATEWAY_TABLENAME	resources	The name if the table containing FHIR resources
DATA_FHIRSERVER_BASEURL	empty string	Can be filled with a disired FHIR Base URL. Default is set to empty string
DATA_FHIRSERVER_USERNAME	empty string	The user name of FHIR Server
DATA_FHIRSERVER_PASSWORD	empty string	The password of FHIR Server user
DATA_FHIRSERVER_CONNECTIONTIMEOUT	3000	The connection timeout of FHIR Server
DATA_FHIRSERVER_SOCKETTIMEOUT	3000	The socket timeout of FHIR Server
DATA_OMOPCDM_JDBCURL	jdbc:postgresql://localhost:5432/ohdsi	The JDBC URL of the OMOP DB
DATA_OMOPCDM_USERNAME	ohdsi_admin_user	The user name of OMOP
DATA_OMOPCDM_PASSWORD	admin1	The password of OMOP user
DATA_OMOPCDM_SCHEMA	cds_cdm	The schema name, where the data will be saved
DATA_BEGINDATE	1800-01-01	It correspond to the 'last_updated_at' column in FHIR-Gateway
DATA_ENDDATE	2099-12-31	It correspond to the 'last_updated_at' column in FHIR-Gateway
PROMETHEUS_PUSHGATEWAY_ENABLED	true	Set to false to disable the prometheus pushgatway
PROMETHEUS_PUSHGATEWAY_URL	https://localhost:9091	The URL of a Prometheus Pushgateway instance
FHIR_SYSTEMS_DEPARTMENT	https://fhir.miracum.org/core/CodeSystem/fachabteilungen	The FHIR identifier system for departments
FHIR_SYSTEMS_INTERPRETATION	http://terminology.hl7.org/CodeSystem/v3-ObservationInterpretation	The system URL for Interpretation
APP_BULKLOAD_ENABLED	false	Set to true to start bulk load
APP_DICTIONARYLOADINRAM_ENABLED	true	Set to false to search data in OMOP DB
APP_WRITEMEDICATIONSTATEMENT_ENABLED	false	Set to true to write MedicationStatement resources to OMOP
APP_STARTSINGLESTEP	emtpy string	Set the desired step name to run the steps separately. Default is the initial load for all FHIR resources
APP_AUTHMODE	none	Set the authentication mode: none, basic or keycloak
APP_KEYCLOAK_SERVERURL	http://keycloak:8080/auth	Set the URL of the Keycloak instance
APP_KEYCLOAK_REALM	blaze	Set the realm name for the FHIR server in Keycloak
APP_KEYCLOAK_CLIENTID	fhir-to-omop	Set the client id within the realm for FHIR server in Keycloak
APP_KEYCLOAK_CLIENTSECRET	secret	Set the credential information of the client
SPRING_CACHE_CAFFEINE_SPEC_MAXIMUMSIZE	5000	The size of the caffeineCache

Development

Install Pre-commit hook to automatically check files on each commit

See https://pre-commit.com for more information

pre-commit install
pre-commit install --hook-type commit-msg

Vocabulary adjustments

csv files for ATC-GM

The OHDSI standardized vocabularies contain only the WHO version of the ATC vocabulary. In Germany, a German Modification of the ATC-WHO exists, which is currently not included in the OHDSI standardized vocabularies. As a consequence, we have manually prepared the ATC-GM vocabulary for use in OMOP CDM. The original source of the ATC-GM is "der amtliche ATC-Index mit DDD-Angaben: GKV-Arzneimittelindex im Wissenschaftlichen Institut der AOK (WIdO), AOK Bundesverband GbR" Version 08.11.2023. For the ATC-GM vocabulary, imports are included for the OMOP CDM tables vocabulary, concept and concept_relationship.