Skip to content

Changes

Page history
Update Incremental Load authored by Henke, Elisa's avatar Henke, Elisa
Show whitespace changes
Inline Side-by-side
Incremental-Load.md
View page @ fe09f641
......@@ -12,7 +12,7 @@ When the job runs as incremental load, all resources are processed in a random o
<summary> Show data flow of Patient resources </summary>
---
When a Patient resource is read in, the job first checks whether this Patient resource already exists in OMOP CDM. This is done in the FHIR_ID_TO_OMOP_ID_MAP table using the logical_id and/or the identifier. If the Patient resource already exists in OMOP CDM, an update will take place in OMOP CDM based on the corresponding person_id (= omop_id from FHIR_ID_TO_OMOP_ID_MAP). If the Patient resource does not yet exist in OMOP CDM, it will be written to OMOP CDM as a new resource.
When a Patient resource is read in, the job first checks whether this Patient resource already exists in OMOP CDM. This is done in the FHIR_ID_TO_OMOP_ID_MAP table using the `logical_id` and/or the `identifier`. If the Patient resource already exists in OMOP CDM, an update will take place in OMOP CDM based on the corresponding `person_id` (= `omop_id` from FHIR_ID_TO_OMOP_ID_MAP). If the Patient resource does not yet exist in OMOP CDM, it will be written to OMOP CDM as a new resource.
![inkrementellesLaden-Patient](uploads/58d4c826a2220a50e2c2b866cdf55949/inkrementellesLaden-Patient.png)
......@@ -25,11 +25,11 @@ When a Patient resource is read in, the job first checks whether this Patient re
<summary> Show data flow of Encounter resources </summary>
---
When a Encounter resource is read in, the job first checks whether the referenced Patient resource already exists in OMOP CDM. This is done in the FHIR_ID_TO_OMOP_ID_MAP table using the logical_id and/or the identifier. If the referenced Patient resource already exists in OMOP CDM, the existing person_id will be used (= omop_id from FHIR_ID_TO_OMOP_ID_MAP).
When a Encounter resource is read in, the job first checks whether the referenced Patient resource already exists in OMOP CDM. This is done in the FHIR_ID_TO_OMOP_ID_MAP table using the `logical_id` and/or the `identifier`. If the referenced Patient resource already exists in OMOP CDM, the existing `person_id` will be used (= `omop_id` from FHIR_ID_TO_OMOP_ID_MAP).
Due to the fact that the resource types are read in random order, it may happen that the referenced Patient resource is not yet available in OMOP CDM. In this case, a dummy for this Patient resource is created and written to OMOP CDM. If the "real" Patient resource is to be written to OMOP CDM during incremental loading, only an update of the dummy takes place.
Next, the job checks if the Encounter resource already exists in OMOP CDM. This is done in the FHIR_ID_TO_OMOP_ID_MAP table using the logical_id and/or the identifier. If the Encounter resource does not yet exist in OMOP CDM, it will be written to OMOP CDM as a new resource. If the Encounter resource already exists in OMOP CDM, an update will take place in OMOP CDM based on the corresponding visit_occurrence_id (= omop_id from FHIR_ID_TO_OMOP_ID_MAP).
Next, the job checks if the Encounter resource already exists in OMOP CDM. This is done in the FHIR_ID_TO_OMOP_ID_MAP table using the `logical_id` and/or the `identifier`. If the Encounter resource does not yet exist in OMOP CDM, it will be written to OMOP CDM as a new resource. If the Encounter resource already exists in OMOP CDM, an update will take place in OMOP CDM based on the corresponding `visit_occurrence_id` (= `omop_id` from FHIR_ID_TO_OMOP_ID_MAP).
![inkrementellesLaden-Encounter](uploads/baec731d4a07c5ca816485a2f7b3d8d6/inkrementellesLaden-Encounter.png)
......@@ -44,13 +44,13 @@ Next, the job checks if the Encounter resource already exists in OMOP CDM. This
---
### Medication
When a Medication resource is read in, the job first checks whether this Medication resource already exists in OMOP CDM. This is done in the FHIR_ID_TO_OMOP_ID_MAP table using the logical_id and/or the identifier. If the Medication resource does not yet exist in OMOP CDM, it will be written to OMOP CDM as a new resource. If the Medication resource already exists in OMOP CDM, an update will take place in FHIR_ID_TO_OMOP_ID_MAP.
When a Medication resource is read in, the job first checks whether this Medication resource already exists in OMOP CDM. This is done in the FHIR_ID_TO_OMOP_ID_MAP table using the `logical_id` and/or the `identifier`. If the Medication resource does not yet exist in OMOP CDM, it will be written to OMOP CDM as a new resource. If the Medication resource already exists in OMOP CDM, an update will take place in FHIR_ID_TO_OMOP_ID_MAP.
### MedicationAdministration
When a MedicationAdministration resource is read in, the job first checks whether the referenced Medication resource already exists in OMOP CDM. This is done in the FHIR_ID_TO_OMOP_ID_MAP table using the logical_id and/or the identifier. If the referenced Medication resource already exists in OMOP CDM, the ATC-code in omop_table is used for further processing of the MedicationAdministration resource.
When a MedicationAdministration resource is read in, the job first checks whether the referenced Medication resource already exists in OMOP CDM. This is done in the FHIR_ID_TO_OMOP_ID_MAP table using the `logical_id` and/or the `identifier`. If the referenced Medication resource already exists in OMOP CDM, the ATC-code in `omop_table` is used for further processing of the MedicationAdministration resource.
Due to the fact that the resource types are read in random order, it may happen that the referenced Medication resource is not yet available in OMOP CDM. In this case, the Medication reference of the MedicationAdministration resource is set as drug_source_value in DRUG_EXPOSURE. After all resources have been processed, post-processing takes place. During post processing, the referenced Medication resource is searched in the FHIR_ID_TO_OMOP_ID_MAP table using the Medication reference in drug_source_value. If this Medication resource exists in OMOP CDM, an update in DRUG_EXPOSURE takes place.
Due to the fact that the resource types are read in random order, it may happen that the referenced Medication resource is not yet available in OMOP CDM. In this case, the Medication reference of the MedicationAdministration resource is set as `drug_source_value` in DRUG_EXPOSURE. After all resources have been processed, post-processing takes place. During post processing, the referenced Medication resource is searched in the FHIR_ID_TO_OMOP_ID_MAP table using the Medication reference in `drug_source_value`. If this Medication resource exists in OMOP CDM, an update in DRUG_EXPOSURE takes place.
![inkrementellesLaden-Medication](uploads/c69677038125a4b6afed204e4e49cddc/inkrementellesLaden-Medication.png)
......@@ -63,7 +63,7 @@ Due to the fact that the resource types are read in random order, it may happen
<summary> Show data flow of Encounter, Procedure, Observation, MedicationAdministration and Condition resources </summary>
---
When a Encounter/Procedure/Observation/MedicationAdministration/Condition resource is read in, the job first checks whether the referenced Patient resource and Encounter resource already exists in OMOP CDM. This is done in the FHIR_ID_TO_OMOP_ID_MAP table using the logical_id and/or the identifier. If the referenced Patient resource and Encounter resource already exists in OMOP CDM, the existing person_id and visit_occurrence_id will be used (= omop_id from FHIR_ID_TO_OMOP_ID_MAP).
When a Encounter/Procedure/Observation/MedicationAdministration/Condition resource is read in, the job first checks whether the referenced Patient resource and Encounter resource already exists in OMOP CDM. This is done in the FHIR_ID_TO_OMOP_ID_MAP table using the `logical_id` and/or the `identifier`. If the referenced Patient resource and Encounter resource already exists in OMOP CDM, the existing `person_id` and `visit_occurrence_id` will be used (= `omop_id` from FHIR_ID_TO_OMOP_ID_MAP).
Due to the fact that the resource types are read in random order, it may happen that the referenced Patient resource and Encounter resource is not yet available in OMOP CDM. In this case, a dummy for the Patient resource and a dummy for the Encounter resource is created and written to OMOP CDM. If the "real" Patient resource and Encounter resource is to be written to OMOP CDM during incremental loading, only an update of the dummies takes place.
......@@ -77,7 +77,7 @@ Next the job checks if the read resource (e.g. Condition) already exists in OMOP
| MedicationAdministration | DRUG_EXPOSURE | person_id <br> drug_concept_id <br> drug_exposure_start_datetime <br> drug_source_concept_id |
| Condition | CONDITION_OCCURRENCE <br> OBSERVATION <br> PROCEDURE_OCCURRENCE <br> MEASUREMENT | person_id <br> condition/observation/procedure/measurement_concept_id <br> condition_start_datetime / observation/procedure/measurement_datetime <br> condition/observation/procedure/measurement_source_concept_id |
If the read resource does not yet exist in OMOP CDM, it will be written to OMOP CDM as a new resource. If the read resource already exists in OMOP CDM, an update will take place in OMOP CDM based on the corresponding visit_detail_id/procedure_occurrence_id/observation_id/measurement_id/drug_exposure_id/condition_occurrence_id.
If the read resource does not yet exist in OMOP CDM, it will be written to OMOP CDM as a new resource. If the read resource already exists in OMOP CDM, an update will take place in OMOP CDM based on the corresponding `visit_detail_id`/`procedure_occurrence_id`/`observation_id`/`measurement_id`/`drug_exposure_id`/`condition_occurrence_id`.
![inkrementellesLaden-Others](uploads/9c72e7aa867269a57cce9e9bc74495de/inkrementellesLaden-Others.png)
......