Data Lineage Report

Executive Summary

MEDIUM Findings

1. Orphaned Dataset: PatientRawParquetLarge

• ID: LIN-001
• Severity: MEDIUM (55)
• Confidence: high
• Rule: ORPHANED_TABLE
• Location: artifacts/dataset/PatientRawParquetLarge.json

Evidence:

ds:PatientRawParquetLarge --[uses_service]--> ls:StorageLS

Description: Dataset PatientRawParquetLarge exists in the environment but has no pipeline or activity that writes to it. The dataset is connected to StorageLS but appears in no pipeline outputs.

Explanation: The lineage graph shows PatientRawParquetLarge connected only via uses_service to StorageLS, with no writes_to edges from any activity. All other Patient-related parquet datasets (PatientAddressParquetLarge, PatientIdentifierParquetLarge, PatientExtensionParquetLarge) are consumed by pipelines, but PatientRawParquetLarge has no consumer or producer.

Remediation: Identify the intended producer pipeline for PatientRawParquetLarge or remove the dataset definition if it is no longer needed. Verify if this dataset should be written by FHIR_Pipeline4Patient_DataFlow_OC or another pipeline.

• Effort: medium

2. Orphaned Dataset: PatientExtensionParquetLarge

• ID: LIN-002
• Severity: MEDIUM (55)
• Confidence: high
• Rule: ORPHANED_TABLE
• Location: artifacts/dataset/PatientExtensionParquetLarge.json

Evidence:

ds:PatientExtensionParquetLarge --[uses_service]--> ls:StorageLS

Description: Dataset PatientExtensionParquetLarge exists in the environment but has no pipeline or activity that writes to it or reads from it. The dataset is connected to StorageLS but appears in no pipeline inputs or outputs.

Explanation: The lineage graph shows PatientExtensionParquetLarge connected only via uses_service to StorageLS, with no reads_from or writes_to edges from any activity. This dataset appears completely disconnected from the data processing workflows.

Remediation: Identify the intended producer and consumer pipelines for PatientExtensionParquetLarge or remove the dataset definition if it is no longer needed. Verify if this dataset should be part of the Patient data processing workflow.

• Effort: medium

3. Undocumented Data Flow: LakeDatabase And Table Creation

• ID: LIN-009
• Severity: MEDIUM (50)
• Confidence: high
• Rule: UNDOCUMENTED_LINEAGE
• Location: artifacts/pipeline/FHIR_Pipeline4Claim_Spark_OC.json:415-470

Evidence:

Activity: LakeDatabase And Table Creation (SynapseNotebook) | deps=[ClaimParquetFlatten_Large] in=[none] out=[none]

Description: Activity LakeDatabase And Table Creation in pipeline FHIR_Pipeline4Claim_Spark_OC has no declared input or output datasets. The notebook likely creates lake database tables but these outputs are not tracked in the lineage graph.

Explanation: The activity shows in=[none] out=[none], indicating that the lake database and table creation operations are not formally tracked in the pipeline lineage. This creates a governance gap for the lake database artifacts.

Remediation: Update the pipeline definition to explicitly declare output datasets or tables for the LakeDatabase And Table Creation activity. This will enable proper lineage tracking for the lake database artifacts.

• Effort: medium

4. Missing Upstream Source: ObservationMain_LargeParquet

• ID: LIN-010
• Severity: MEDIUM (60)
• Confidence: medium
• Rule: MISSING_UPSTREAM
• Location: artifacts/dataset/ObservationMain_LargeParquet.json

Evidence:

ds:ObservationMain_LargeParquet --[reads_from]--> act:FHIR_Pipeline4Observation_Spark_OC:Observation_Parquet_large2SQL

Description: Dataset ObservationMain_LargeParquet is consumed by activity Observation_Parquet_large2SQL but has no explicit producer activity in the lineage graph. The dataset is likely produced by ObservationParquetFlatten_Large notebook but this relationship is not formally declared.

Explanation: The lineage graph shows ObservationMain_LargeParquet being read by a Copy activity, but there is no writes_to edge from any activity to this dataset. The implicit producer is the ObservationParquetFlatten_Large notebook based on naming patterns and pipeline structure.

Remediation: Update the FHIR_Pipeline4Observation_Spark_OC pipeline to explicitly declare ObservationMain_LargeParquet as an output of the ObservationParquetFlatten_Large activity. This will establish formal lineage between the producer and consumer.

• Effort: low

5. Missing Upstream Source: PatientAddressParquetLarge

• ID: LIN-011
• Severity: MEDIUM (60)
• Confidence: medium
• Rule: MISSING_UPSTREAM
• Location: artifacts/dataset/PatientAddressParquetLarge.json

Evidence:

ds:PatientAddressParquetLarge --[reads_from]--> act:FHIR_Pipeline4Patient_DataFlow_OC:PatientAddress_large2SQL

Description: Dataset PatientAddressParquetLarge is consumed by activity PatientAddress_large2SQL but has no explicit producer activity in the lineage graph. The dataset is likely produced by PatientParquet2Sink data flow but this relationship is not formally declared.

Explanation: The lineage graph shows PatientAddressParquetLarge being read by a Copy activity, but there is no writes_to edge from any activity to this dataset. The implicit producer is the PatientParquet2Sink data flow based on naming patterns and pipeline structure.

Remediation: Update the FHIR_Pipeline4Patient_DataFlow_OC pipeline to explicitly declare PatientAddressParquetLarge as an output of the PatientParquet2Sink activity. This will establish formal lineage between the producer and consumer.

• Effort: low

6. Missing Upstream Source: PatientIdentifierParquetLarge

• ID: LIN-012
• Severity: MEDIUM (60)
• Confidence: medium
• Rule: MISSING_UPSTREAM
• Location: artifacts/dataset/PatientIdentifierParquetLarge.json

Evidence:

ds:PatientIdentifierParquetLarge --[reads_from]--> act:FHIR_Pipeline4Patient_DataFlow_OC:PatientIdentifier_large2SQL

Description: Dataset PatientIdentifierParquetLarge is consumed by activity PatientIdentifier_large2SQL but has no explicit producer activity in the lineage graph. The dataset is likely produced by PatientParquet2Sink data flow but this relationship is not formally declared.

Explanation: The lineage graph shows PatientIdentifierParquetLarge being read by a Copy activity, but there is no writes_to edge from any activity to this dataset. The implicit producer is the PatientParquet2Sink data flow based on naming patterns and pipeline structure.

Remediation: Update the FHIR_Pipeline4Patient_DataFlow_OC pipeline to explicitly declare PatientIdentifierParquetLarge as an output of the PatientParquet2Sink activity. This will establish formal lineage between the producer and consumer.

• Effort: low

7. Missing Upstream Source: ClaimDiagnosisParquetLarge

• ID: LIN-013
• Severity: MEDIUM (60)
• Confidence: medium
• Rule: MISSING_UPSTREAM
• Location: artifacts/dataset/ClaimDiagnosisParquetLarge.json

Evidence:

ds:ClaimDiagnosisParquetLarge --[reads_from]--> act:FHIR_Pipeline4Claim_Spark_OC:ClaimDiagnosis2SQL

Description: Dataset ClaimDiagnosisParquetLarge is consumed by activity ClaimDiagnosis2SQL but has no explicit producer activity in the lineage graph. The dataset is likely produced by ClaimParquetFlatten_Large notebook but this relationship is not formally declared.

Explanation: The lineage graph shows ClaimDiagnosisParquetLarge being read by a Copy activity, but there is no writes_to edge from any activity to this dataset. The implicit producer is the ClaimParquetFlatten_Large notebook based on naming patterns and pipeline structure.

Remediation: Update the FHIR_Pipeline4Claim_Spark_OC pipeline to explicitly declare ClaimDiagnosisParquetLarge as an output of the ClaimParquetFlatten_Large activity. This will establish formal lineage between the producer and consumer.

• Effort: low

8. Missing Upstream Source: ClaimInsuranceParquetLarge

• ID: LIN-014
• Severity: MEDIUM (60)
• Confidence: medium
• Rule: MISSING_UPSTREAM
• Location: artifacts/dataset/ClaimInsuranceParquetLarge.json

Evidence:

ds:ClaimInsuranceParquetLarge --[reads_from]--> act:FHIR_Pipeline4Claim_Spark_OC:ClaimInsurance2SQL

Description: Dataset ClaimInsuranceParquetLarge is consumed by activity ClaimInsurance2SQL but has no explicit producer activity in the lineage graph. The dataset is likely produced by ClaimParquetFlatten_Large notebook but this relationship is not formally declared.

Explanation: The lineage graph shows ClaimInsuranceParquetLarge being read by a Copy activity, but there is no writes_to edge from any activity to this dataset. The implicit producer is the ClaimParquetFlatten_Large notebook based on naming patterns and pipeline structure.

Remediation: Update the FHIR_Pipeline4Claim_Spark_OC pipeline to explicitly declare ClaimInsuranceParquetLarge as an output of the ClaimParquetFlatten_Large activity. This will establish formal lineage between the producer and consumer.

• Effort: low

9. Missing Upstream Source: ClaimProcedureParquetLarge

• ID: LIN-015
• Severity: MEDIUM (60)
• Confidence: medium
• Rule: MISSING_UPSTREAM
• Location: artifacts/dataset/ClaimProcedureParquetLarge.json

Evidence:

ds:ClaimProcedureParquetLarge --[reads_from]--> act:FHIR_Pipeline4Claim_Spark_OC:ClaimProcedure2SQL

Description: Dataset ClaimProcedureParquetLarge is consumed by activity ClaimProcedure2SQL but has no explicit producer activity in the lineage graph. The dataset is likely produced by ClaimParquetFlatten_Large notebook but this relationship is not formally declared.

Explanation: The lineage graph shows ClaimProcedureParquetLarge being read by a Copy activity, but there is no writes_to edge from any activity to this dataset. The implicit producer is the ClaimParquetFlatten_Large notebook based on naming patterns and pipeline structure.

Remediation: Update the FHIR_Pipeline4Claim_Spark_OC pipeline to explicitly declare ClaimProcedureParquetLarge as an output of the ClaimParquetFlatten_Large activity. This will establish formal lineage between the producer and consumer.

• Effort: low

Remediation Roadmap

Long-term (1-3 months)

• LIN-001: Orphaned Dataset: PatientRawParquetLarge — Identify the intended producer pipeline for PatientRawParquetLarge or remove the dataset definition if it is no longer needed. Verify if this dataset should be written by FHIR_Pipeline4Patient_DataFlow_OC or another pipeline.
• LIN-002: Orphaned Dataset: PatientExtensionParquetLarge — Identify the intended producer and consumer pipelines for PatientExtensionParquetLarge or remove the dataset definition if it is no longer needed. Verify if this dataset should be part of the Patient data processing workflow.
• LIN-009: Undocumented Data Flow: LakeDatabase And Table Creation — Update the pipeline definition to explicitly declare output datasets or tables for the LakeDatabase And Table Creation activity. This will enable proper lineage tracking for the lake database artifacts.
• LIN-010: Missing Upstream Source: ObservationMain_LargeParquet — Update the FHIR_Pipeline4Observation_Spark_OC pipeline to explicitly declare ObservationMain_LargeParquet as an output of the ObservationParquetFlatten_Large activity. This will establish formal lineage between the producer and consumer.
• LIN-011: Missing Upstream Source: PatientAddressParquetLarge — Update the FHIR_Pipeline4Patient_DataFlow_OC pipeline to explicitly declare PatientAddressParquetLarge as an output of the PatientParquet2Sink activity. This will establish formal lineage between the producer and consumer.
• LIN-012: Missing Upstream Source: PatientIdentifierParquetLarge — Update the FHIR_Pipeline4Patient_DataFlow_OC pipeline to explicitly declare PatientIdentifierParquetLarge as an output of the PatientParquet2Sink activity. This will establish formal lineage between the producer and consumer.
• LIN-013: Missing Upstream Source: ClaimDiagnosisParquetLarge — Update the FHIR_Pipeline4Claim_Spark_OC pipeline to explicitly declare ClaimDiagnosisParquetLarge as an output of the ClaimParquetFlatten_Large activity. This will establish formal lineage between the producer and consumer.
• LIN-014: Missing Upstream Source: ClaimInsuranceParquetLarge — Update the FHIR_Pipeline4Claim_Spark_OC pipeline to explicitly declare ClaimInsuranceParquetLarge as an output of the ClaimParquetFlatten_Large activity. This will establish formal lineage between the producer and consumer.
• LIN-015: Missing Upstream Source: ClaimProcedureParquetLarge — Update the FHIR_Pipeline4Claim_Spark_OC pipeline to explicitly declare ClaimProcedureParquetLarge as an output of the ClaimParquetFlatten_Large activity. This will establish formal lineage between the producer and consumer.