Data Engineer

Senior Data Engineer
Primary Skills:

1. Python, PySpark and SQL

2. Snowflake

3. Medallion Architecture

4. Data Bricks

Position summary
The Senior Data Engineer will build trusted real-time and historical data products for the Real-Time Finance Close and Value-Based Care platform.
The role will integrate claims, encounters, member eligibility, provider, contract, capitation, sub-capitation, actuarial, journal, subledger, GL, payment, and operational data. The engineer will create streaming and batch pipelines that support financial processing, actuarial analysis, reconciliation, Cube.dev semantic models, Power BI reporting, ontology integration, and AI-driven analytics.
The engineer must be able to preserve financial and operational history, correctly process late and retroactive changes, and reproduce what the organization knew at a particular accounting or valuation date.
Key responsibilities
Data ingestion and integration

• Build reliable ingestion pipelines for:
• Claims and claim lines
• Clinical encounters
• Eligibility and enrollment
• Member attribution
• Provider rosters and hierarchies
• Payer and provider contracts
• Capitation and sub-capitation
• Remittances and payments
• Risk adjustment
• Quality and performance measures
• Actuarial model outputs
• Journals, subledgers, and GL balances
• Support event streaming, APIs, change-data capture, files, databases, and partner interfaces.
• Develop reusable patterns for schema evolution, quarantine, reprocessing, and source-system reconciliation.

Streaming and transformation engineering

• Develop stateful and stateless streaming pipelines.
• Correctly handle:
• Event time versus processing time
• Late-arriving data
• Out-of-order events
• Duplicate events
• Windowing and watermarks
• Retroactive eligibility changes
• Reattribution
• Contract amendments
• Claim reversals and adjustments
• Create incremental, restartable, and replay-safe transformations.
• Maintain control totals and processing checkpoints.

Healthcare and financial data modeling

• Design data products for:
• Member-month populations
• Contract and risk arrangement
• Inbound capitation
• Outbound sub-capitation
• Provider settlement
• FFS claims and encounters
• Medical-cost and utilization analysis
• Actuarial reserves
• Journal and GL activity
• Reconciliation exceptions
• Use dimensional, normalized, Data Vault, lakehouse, graph, or other modeling approaches where appropriate.
• Implement effective-dated and bitemporal models.
• Preserve transaction date, service date, event date, accounting date, posting date, valuation date, and reporting period as distinct concepts.
• Support legal entity, market, line of business, product, provider, cost center, and accounting dimensions.

Data quality, lineage, and control

• Build automated controls for completeness, validity, uniqueness, timeliness, consistency, and referential integrity.
• Reconcile source totals to data products, calculations, subledgers, GL balances, and semantic metrics.
• Detect missing, duplicate, delayed, and materially changed records.
• Capture source-to-target lineage and business transformation lineage.
• Provide point-in-time reproducibility for accounting, actuarial, audit, and regulatory review.
• Work with data stewards to define data-quality thresholds, issue ownership, and remediation processes.

Analytics and semantic enablement

• Create curated data products for Cube.dev, Power BI, data science, ontology, and AI consumers.
• Partner with semantic modelers to implement governed definitions for:
• Eligible member months
• Attributed population
• Capitation revenue
• Sub-capitation expense
• Paid, reported, and incurred claims
• IBNR and reserves
• PMPM revenue and cost
• Medical cost ratio
• Provider settlement
• Reconciliation status
• Close status
• Optimize data layouts and aggregations for predictable performance.
• Prevent metric logic from being inconsistently reimplemented across pipelines, dashboards, and AI applications.

Data platform operations

• Build data pipelines through version-controlled, automated deployment processes.
• Monitor freshness, volume, schema, lineage, quality, and processing latency.
• Participate in production support, root-cause analysis, replay, and recovery.
• Apply access controls, encryption, masking, tokenization, and data-retention policies.
• Manage PHI and financial data according to minimum-necessary access principles.

Required qualifications

• Typically 6 or more years of data engineering experience.
• Advanced SQL skills and experience with Python, Scala, Java, or another data engineering language.
• Hands-on experience with distributed processing frameworks such as Spark, Flink, Beam, Kafka Streams, or equivalent.
• Experience with Kafka or another event-streaming platform.
• Experience building both batch and streaming data pipelines.
• Experience with a cloud data platform, lakehouse, data warehouse, or distributed storage environment.
• Strong understanding of dimensional modeling, data partitioning, incremental processing, and performance optimization.
• Experience with data orchestration, automated deployment, and version-controlled data transformation.
• Experience implementing data-quality tests and production data monitoring.
• Understanding of identity, access controls, sensitive-data handling, and secure data engineering.