Diagram 03 · ETL Pipeline

Multi-Source Data Pipeline

Eight stages from raw artifact to curated data product. Orchestrated by Airflow via Astronomer. Lineage through dbt and Snowflake native object dependencies.

CONTROL Airflow via Astronomer · Pipeline Orchestrator · vendor-maintained image on Game Warden routes content type · coordinates structured + unstructured tracks · manages retries · prioritization lanes · cost guardrails · dbt lineage SOURCE ARTIFACTS Cables released artifacts Documents PDF · Office · email Imagery scanned · sat · digital Video UAV · feeds · archives Audio intercepts · debriefs Foreign-Lang docs · transcripts Geospatial KMZ · GeoJSON · WMS Signal Exports SIGINT · ELINT Zip Archives mixed bundles Legacy Migrations DOORS · archives STAGE 1 INTAKE Airbyte: structured ingest in-enclave object store STAGE 2 EXPAND archive expansion zip bomb protection STAGE 3 SCAN malware · DLP type detection STAGE 4 PARSE layout-aware unified elements STAGE 5 MEDIA OCR · ASR · video translation STAGE 6 SEMANTIC NER · class · embed relations STAGE 7 CURATE DQ · MDM merge lineage attached STAGE 8 LAND Snowflake products indexed for search ClamAV · YARA malware detection Commercial DLP PII · markings unstructured.io layout parsing Table · Form Extract key-value · tables Tesseract OCR Whisper speech-to-text Video Intel keyframes · objects Translation Hub LILT · Microsoft Language + NER links to MDM Titus Classification NOFORN · ORCON · REL TO Embeddings Cohere · BGE Relation Extract coreference · graph dbt Tests · DQ confidence scoring Snowflake DQ data metric functions dbt Lineage provenance · object deps SNOWFLAKE data products cables reports entities transcripts imagery embeddings audit PIPELINE GOVERNANCE & CONTROLS Cost Guardrails throttle AI calls at budget tiered processing sample before expensive ops Priority Lanes time-critical fast path batch optimization for bulk SLA-aware scheduling Retry · DLQ · Quarantine poison pill quarantine exponential backoff dead letter queue Lineage + Audit every record traces to source audit-grade provenance dbt Lineage + Titus tags CDC → Elastic Snowpipe Streaming near-real-time search index sync