Pipeline_Health CodePrizm

Generated by CodePrizm

Pipeline Health Report

Executive Summary

MetricValue
Overall Score94/100
Total Findings24
Critical0
High4
Medium17
Low3
Info0

HIGH Findings

1. Single Point of Failure: Create Tables Activity

Evidence:

Activity: Create Tables (Script) | deps=[ObservationParquetFlatten_Large] in=[none] out=[none]
    linked_service: SynapseDedicatedPoolLS

Description: The Create Tables activity in FHIR_Pipeline4Observation_Spark_OC is a critical blocking activity that all downstream Copy activities depend on. If this Script activity fails, the entire pipeline is blocked with no failover or redundancy mechanism.

Explanation: This activity blocks downstream Copy activity. The dependency graph shows: ObservationParquetFlatten_Large → Create Tables → Observation_Parquet_large2SQL.

Remediation: Implement error handling with retry logic for the Create Tables activity. Consider adding a validation activity to check if tables already exist before attempting creation.

2. Single Point of Failure: Create Tables Activity

Evidence:

Activity: Create Tables (Script) | deps=[PatientParquet2Sink] in=[none] out=[none]
    linked_service: SynapseDedicatedPoolLS

Description: The Create Tables activity in FHIR_Pipeline4Patient_DataFlow_OC is a critical blocking activity that two downstream Copy activities depend on. If this Script activity fails, PatientAddress_large2SQL and PatientIdentifier_large2SQL cannot execute.

Explanation: This activity blocks two critical Copy activities. The dependency graph shows: PatientParquet2Sink → Create Tables → PatientAddress_large2SQL and PatientIdentifier_large2SQL.

Remediation: Add retry policy with exponential backoff for the Create Tables activity. Implement pre-execution validation to check table existence.

3. Single Point of Failure: Create Tables Activity

Evidence:

Activity: Create Tables (Script) | deps=[ClaimParquetFlatten_Large] in=[none] out=[none]
    linked_service: SynapseDedicatedPoolLS

Description: The Create Tables activity in FHIR_Pipeline4Claim_Spark_OC is a critical blocking activity that three downstream Copy activities depend on. If this Script activity fails, ClaimDiagnosis2SQL, ClaimInsurance2SQL, and ClaimProcedure2SQL cannot execute.

Explanation: This activity blocks three critical Copy activities. The dependency graph shows: ClaimParquetFlatten_Large → Create Tables → ClaimDiagnosis2SQL, ClaimInsurance2SQL, ClaimProcedure2SQL.

Remediation: Add retry policy with exponential backoff for the Create Tables activity. Implement pre-execution validation to check table existence. Add error handling activities.

4. Single Point of Failure: Copy Source Data To Raw Container

Evidence:

Activity: Copy Source Data To Raw Container (Copy) | deps=[none] in=[Source_DataPrep_DS] out=[Sink_DataPrep_DS]

Description: The Copy Source Data To Raw Container activity is a critical blocking activity that two downstream Copy activities depend on. If this activity fails, Create Curated Container and Create Processed Container cannot execute.

Explanation: This activity blocks two downstream activities. The dependency graph shows: Copy Source Data To Raw Container → Create Curated Container and Create Processed Container.

Remediation: Add retry policy for the Copy activity. Implement error handling with alternative execution paths or notification mechanisms.

MEDIUM Findings

5. Missing Retry Policy: Copy Activity

Evidence:

Activity: Observation_Parquet_large2SQL (Copy) | deps=[Create Tables] in=[ObservationMain_LargeParquet] out=[Observation_SQLDS]

Description: The Observation_Parquet_large2SQL Copy activity lacks explicit retry configuration. Copy activities are prone to transient failures when moving data between storage and SQL pools.

Explanation: Copy activities should have retry policies to handle transient network issues, temporary resource unavailability, or throttling from source/sink services.

Remediation: Add retry policy with exponential backoff to the Copy activity. Configure retry count and interval appropriate for data volume and network conditions.

6. Missing Retry Policy: Copy Activity

Evidence:

Activity: PatientAddress_large2SQL (Copy) | deps=[Create Tables] in=[PatientAddressParquetLarge] out=[PatientAddressSQL]

Description: The PatientAddress_large2SQL Copy activity lacks explicit retry configuration. Copy activities are prone to transient failures when moving data between storage and SQL pools.

Explanation: Copy activities should have retry policies to handle transient network issues, temporary resource unavailability, or throttling from source/sink services.

Remediation: Add retry policy with exponential backoff to the Copy activity. Configure retry count and interval appropriate for data volume and network conditions.

7. Missing Retry Policy: Copy Activity

Evidence:

Activity: PatientIdentifier_large2SQL (Copy) | deps=[Create Tables] in=[PatientIdentifierParquetLarge] out=[PatientIdentifierSQLLarge]

Description: The PatientIdentifier_large2SQL Copy activity lacks explicit retry configuration. Copy activities are prone to transient failures when moving data between storage and SQL pools.

Explanation: Copy activities should have retry policies to handle transient network issues, temporary resource unavailability, or throttling from source/sink services.

Remediation: Add retry policy with exponential backoff to the Copy activity. Configure retry count and interval appropriate for data volume and network conditions.

8. Missing Retry Policy: Copy Activity

Evidence:

Activity: ClaimDiagnosis2SQL (Copy) | deps=[Create Tables] in=[ClaimDiagnosisParquetLarge] out=[ClaimDiagnosisSQL]

Description: The ClaimDiagnosis2SQL Copy activity lacks explicit retry configuration. Copy activities are prone to transient failures when moving data between storage and SQL pools.

Explanation: Copy activities should have retry policies to handle transient network issues, temporary resource unavailability, or throttling from source/sink services.

Remediation: Add retry policy with exponential backoff to the Copy activity. Configure retry count and interval appropriate for data volume and network conditions.

9. Missing Retry Policy: Copy Activity

Evidence:

Activity: ClaimInsurance2SQL (Copy) | deps=[Create Tables] in=[ClaimInsuranceParquetLarge] out=[ClaimInsurance]

Description: The ClaimInsurance2SQL Copy activity lacks explicit retry configuration. Copy activities are prone to transient failures when moving data between storage and SQL pools.

Explanation: Copy activities should have retry policies to handle transient network issues, temporary resource unavailability, or throttling from source/sink services.

Remediation: Add retry policy with exponential backoff to the Copy activity. Configure retry count and interval appropriate for data volume and network conditions.

10. Missing Retry Policy: Copy Activity

Evidence:

Activity: ClaimProcedure2SQL (Copy) | deps=[Create Tables] in=[ClaimProcedureParquetLarge] out=[ClaimProcedureSQL]

Description: The ClaimProcedure2SQL Copy activity lacks explicit retry configuration. Copy activities are prone to transient failures when moving data between storage and SQL pools.

Explanation: Copy activities should have retry policies to handle transient network issues, temporary resource unavailability, or throttling from source/sink services.

Remediation: Add retry policy with exponential backoff to the Copy activity. Configure retry count and interval appropriate for data volume and network conditions.

11. Missing Retry Policy: Copy Activity

Evidence:

Activity: Copy Source Data To Raw Container (Copy) | deps=[none] in=[Source_DataPrep_DS] out=[Sink_DataPrep_DS]

Description: The Copy Source Data To Raw Container Copy activity lacks explicit retry configuration. Copy activities are prone to transient failures when moving data between storage accounts.

Explanation: Copy activities should have retry policies to handle transient network issues, temporary resource unavailability, or throttling from source/sink services.

Remediation: Add retry policy with exponential backoff to the Copy activity. Configure retry count and interval appropriate for data volume and network conditions.

12. Missing Retry Policy: Copy Activity

Evidence:

Activity: Create Curated Container (Copy) | deps=[Copy Source Data To Raw Container] in=[Source_DataPrep_Curated_DS] out=[Sink_DataPrep_Curated_DS]

Description: The Create Curated Container Copy activity lacks explicit retry configuration. Copy activities are prone to transient failures when moving data between storage accounts.

Explanation: Copy activities should have retry policies to handle transient network issues, temporary resource unavailability, or throttling from source/sink services.

Remediation: Add retry policy with exponential backoff to the Copy activity. Configure retry count and interval appropriate for data volume and network conditions.

13. Missing Retry Policy: Copy Activity

Evidence:

Activity: Create Processed Container (Copy) | deps=[Copy Source Data To Raw Container] in=[Source_DataPrep_Processed_DS] out=[Sink_DataPrep_Processed_DS]

Description: The Create Processed Container Copy activity lacks explicit retry configuration. Copy activities are prone to transient failures when moving data between storage accounts.

Explanation: Copy activities should have retry policies to handle transient network issues, temporary resource unavailability, or throttling from source/sink services.

Remediation: Add retry policy with exponential backoff to the Copy activity. Configure retry count and interval appropriate for data volume and network conditions.

14. Missing Error Handling: Pipeline Level

Evidence:

FHIR_Pipeline4Observation_Spark_OC (artifacts/pipeline/FHIR_Pipeline4Observation_Spark_OC.json) — 4 activities

Description: The FHIR_Pipeline4Observation_Spark_OC pipeline lacks error handling activities or failure paths. No on-failure activities, alert hooks, or notification mechanisms are configured.

Explanation: Without error handling, pipeline failures are silent and may go unnoticed until downstream systems report missing data. Critical activities like Create Tables should have explicit failure paths.

Remediation: Add error handling activities such as Web activity to send alerts, or Execute Pipeline activity to trigger error handling workflows. Configure on-failure dependencies for critical activities.

15. Missing Error Handling: Pipeline Level

Evidence:

FHIR_Pipeline4Patient_DataFlow_OC (artifacts/pipeline/FHIR_Pipeline4Patient_DataFlow_OC.json) — 5 activities

Description: The FHIR_Pipeline4Patient_DataFlow_OC pipeline lacks error handling activities or failure paths. No on-failure activities, alert hooks, or notification mechanisms are configured.

Explanation: Without error handling, pipeline failures are silent and may go unnoticed until downstream systems report missing data. Critical activities like Create Tables should have explicit failure paths.

Remediation: Add error handling activities such as Web activity to send alerts, or Execute Pipeline activity to trigger error handling workflows. Configure on-failure dependencies for critical activities.

16. Missing Error Handling: Pipeline Level

Evidence:

FHIR_Pipeline4Claim_Spark_OC (artifacts/pipeline/FHIR_Pipeline4Claim_Spark_OC.json) — 7 activities

Description: The FHIR_Pipeline4Claim_Spark_OC pipeline lacks error handling activities or failure paths. No on-failure activities, alert hooks, or notification mechanisms are configured.

Explanation: Without error handling, pipeline failures are silent and may go unnoticed until downstream systems report missing data. Critical activities like Create Tables should have explicit failure paths.

Remediation: Add error handling activities such as Web activity to send alerts, or Execute Pipeline activity to trigger error handling workflows. Configure on-failure dependencies for critical activities.

17. Missing Error Handling: Pipeline Level

Evidence:

Copy_Data_Source_To_Raw_PL (artifacts/pipeline/Copy_Data_Source_To_Raw_PL.json) — 3 activities

Description: The Copy_Data_Source_To_Raw_PL pipeline lacks error handling activities or failure paths. No on-failure activities, alert hooks, or notification mechanisms are configured.

Explanation: Without error handling, pipeline failures are silent and may go unnoticed until downstream systems report missing data. The initial Copy activity is critical and should have explicit failure paths.

Remediation: Add error handling activities such as Web activity to send alerts, or Execute Pipeline activity to trigger error handling workflows. Configure on-failure dependencies for critical activities.

18. No Monitoring or Alerting Configured

Evidence:

FHIR_Pipeline4Observation_Spark_OC (artifacts/pipeline/FHIR_Pipeline4Observation_Spark_OC.json) — 4 activities

Description: The FHIR_Pipeline4Observation_Spark_OC pipeline lacks monitoring or alerting configuration. No health-check or heartbeat mechanisms are visible in the pipeline definition.

Explanation: Without monitoring, pipeline health issues may go undetected. Critical pipelines should have alerting configured for failures and performance degradation.

Remediation: Configure Azure Monitor alerts for pipeline failures, long-running executions, and activity failures. Add logging activities to track pipeline execution metrics.

19. No Monitoring or Alerting Configured

Evidence:

FHIR_Pipeline4Patient_DataFlow_OC (artifacts/pipeline/FHIR_Pipeline4Patient_DataFlow_OC.json) — 5 activities

Description: The FHIR_Pipeline4Patient_DataFlow_OC pipeline lacks monitoring or alerting configuration. No health-check or heartbeat mechanisms are visible in the pipeline definition.

Explanation: Without monitoring, pipeline health issues may go undetected. Critical pipelines should have alerting configured for failures and performance degradation.

Remediation: Configure Azure Monitor alerts for pipeline failures, long-running executions, and activity failures. Add logging activities to track pipeline execution metrics.

20. No Monitoring or Alerting Configured

Evidence:

FHIR_Pipeline4Claim_Spark_OC (artifacts/pipeline/FHIR_Pipeline4Claim_Spark_OC.json) — 7 activities

Description: The FHIR_Pipeline4Claim_Spark_OC pipeline lacks monitoring or alerting configuration. No health-check or heartbeat mechanisms are visible in the pipeline definition.

Explanation: Without monitoring, pipeline health issues may go undetected. Critical pipelines should have alerting configured for failures and performance degradation.

Remediation: Configure Azure Monitor alerts for pipeline failures, long-running executions, and activity failures. Add logging activities to track pipeline execution metrics.

21. No Monitoring or Alerting Configured

Evidence:

Copy_Data_Source_To_Raw_PL (artifacts/pipeline/Copy_Data_Source_To_Raw_PL.json) — 3 activities

Description: The Copy_Data_Source_To_Raw_PL pipeline lacks monitoring or alerting configuration. No health-check or heartbeat mechanisms are visible in the pipeline definition.

Explanation: Without monitoring, pipeline health issues may go undetected. Critical pipelines should have alerting configured for failures and performance degradation.

Remediation: Configure Azure Monitor alerts for pipeline failures, long-running executions, and activity failures. Add logging activities to track pipeline execution metrics.

LOW Findings

22. Long Activity Dependency Chain

Evidence:

NDJSON_Ingestion_Observation → ObservationParquetFlatten_Large
  ObservationParquetFlatten_Large → Create Tables
  Create Tables → Observation_Parquet_large2SQL

Description: The FHIR_Pipeline4Observation_Spark_OC pipeline has a dependency chain of 4 activities. Long chains amplify failure blast radius where early failures cascade through the entire pipeline.

Explanation: The chain NDJSON_Ingestion_Observation → ObservationParquetFlatten_Large → Create Tables → Observation_Parquet_large2SQL creates a sequential dependency where any failure blocks all downstream work.

Remediation: Consider parallelizing independent activities where possible. Add checkpointing or intermediate validation steps to enable partial recovery.

23. Long Activity Dependency Chain

Evidence:

NDJSON_Ingestion_Patient → PatientParquet2Sink
  PatientParquet2Sink → Create Tables
  Create Tables → PatientAddress_large2SQL
  Create Tables → PatientIdentifier_large2SQL

Description: The FHIR_Pipeline4Patient_DataFlow_OC pipeline has a dependency chain of 4 activities with branching. Long chains amplify failure blast radius where early failures cascade through the entire pipeline.

Explanation: The chain NDJSON_Ingestion_Patient → PatientParquet2Sink → Create Tables → [PatientAddress_large2SQL, PatientIdentifier_large2SQL] creates a sequential dependency where any failure blocks all downstream work.

Remediation: Consider parallelizing independent activities where possible. Add checkpointing or intermediate validation steps to enable partial recovery.

24. Long Activity Dependency Chain

Evidence:

NDJSON_Ingestion_Claim → ClaimParquetFlatten_Large
  ClaimParquetFlatten_Large → Create Tables
  Create Tables → ClaimDiagnosis2SQL
  Create Tables → ClaimInsurance2SQL
  Create Tables → ClaimProcedure2SQL
  ClaimParquetFlatten_Large → LakeDatabase And Table Creation

Description: The FHIR_Pipeline4Claim_Spark_OC pipeline has a dependency chain of 4 activities with multiple branches. Long chains amplify failure blast radius where early failures cascade through the entire pipeline.

Explanation: The chain NDJSON_Ingestion_Claim → ClaimParquetFlatten_Large → Create Tables → [ClaimDiagnosis2SQL, ClaimInsurance2SQL, ClaimProcedure2SQL] creates a sequential dependency where any failure blocks all downstream work.

Remediation: Consider parallelizing independent activities where possible. Add checkpointing or intermediate validation steps to enable partial recovery.

Remediation Roadmap

Short-term (1-4 weeks)

Long-term (1-3 months)