Data Engineer – Analytic Platform & Data Pipelines

Role Overview

3GIMBALS is seeking a Data Engineer to design, build, and maintain the data pipelines and infrastructure that power our unclassified PAI/CAI-based analytic platform. This role is responsible for ingesting, transforming, and curating large volumes of structured and unstructured data from diverse open and commercial sources; building resilient, automated ETL/ELT workflows; and ensuring data is high-quality, well-governed, and analysis-ready for the downstream analytics, knowledge graph, and modeling teams. The ideal candidate is comfortable working with messy, multi-source data at scale within secure development environments.

Key Responsibilities

Data Pipeline Development & Ingestion

• Design and build scalable batch and streaming pipelines to ingest structured and unstructured data from PAI/CAI sources, APIs, and third-party feeds
• Develop ETL/ELT workflows to normalize, enrich, and transform heterogeneous data into standardized schemas
• Build and maintain automated ingestion connectors for web, document, geospatial, and tabular data sources
• Manage data orchestration and scheduling using tools such as Airflow, Dagster, or Prefect

Data Modeling & Storage

• Design and maintain data models, schemas, and storage layers across relational, NoSQL, and object stores
• Build and maintain data lakes/lakehouses and curated, analysis-ready data marts
• Optimize partitioning, indexing, and query performance for large datasets
• Support entity resolution and data linking in coordination with the knowledge graph and modeling teams

Data Quality, Governance & Lineage

• Implement data validation, quality checks, and monitoring across pipelines
• Establish data lineage, cataloging, and metadata management
• Enforce data governance, provenance tracking, and source attribution appropriate for PAI/CAI data
• Document datasets, schemas, and pipeline logic for downstream consumers

Security & Compliance

• Ensure pipelines and data stores meet security requirements for operation in sensitive environments
• Implement encryption, access control, and secure data-handling practices
• Support Authority to Operate (ATO) processes and compliance frameworks

Required Qualifications

Technical Expertise

• 4+ years of data engineering experience building and operating production data pipelines
• Strong programming skills in Python and SQL (Scala or Java a plus)
• Experience with distributed data processing frameworks (Spark, Dask, or similar)
• Hands-on experience with workflow orchestration tools (Airflow, Dagster, Prefect)
• Proficiency with relational and NoSQL databases (PostgreSQL, MongoDB, Elasticsearch, etc.)
• Experience with cloud data platforms and services (AWS, Azure, or GCP)

Data & Infrastructure

• Experience designing data models, warehouses, and lakehouse architectures
• Familiarity with data formats and serialization (Parquet, Avro, JSON, GeoJSON)
• Understanding of data quality, lineage, and governance practices
• Experience with containerization (Docker) and CI/CD for data workflows

Domain Knowledge

• Experience working with large-scale, heterogeneous, or open-source datasets
• Understanding of data provenance and source-attribution requirements

Preferred Qualifications

• Active security clearance or ability to obtain one
• Experience in government, defense, or intelligence contracting environments
• Familiarity with PAI/CAI (publicly and commercially available information) data sources
• Experience with geospatial data processing (PostGIS, GDAL, or similar)
• Knowledge of graph data structures and preparing data for knowledge graphs
• Experience with streaming platforms (Kafka, Kinesis)
• Familiarity with federal compliance frameworks (FedRAMP, FISMA, NIST 800-53)

Technical Environment

• Languages: Python, SQL (Scala/Java a plus)
• Processing: Spark, Airflow/Dagster/Prefect, streaming frameworks
• Storage: PostgreSQL, Elasticsearch, object storage / data lake, Parquet
• Infrastructure: Docker, Kubernetes, cloud platforms (AWS GovCloud, Azure Government)
• Security: Encryption at rest and in transit, RBAC, secure data handling

This role is central to the platform: the data engineering team delivers the clean, trustworthy, well-documented data that every analytic, knowledge graph, and risk-modeling capability depends on.