Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

Aarista Technologies

NewBe an early applicant

Senior Databricks Engineer - Revenue Cycle Management (RCM)

Posted

Description

About the Role


We are seeking a Senior Databricks Engineer to design, build, and optimize large-scale data pipelines powering our healthcare Revenue Cycle Management platform. You will own end-to-end data engineering across EDI claim processing (837, 835, 277, 999), remittance workflows, AI-driven medical coding, and real-time operational data delivery using the Medallion architecture on Azure Databricks.


Key Responsibilities:


  • Design & maintain production data pipelines using PySpark and Scala on Azure Databricks, processing healthcare EDI transactions (837 Claims, 835 Remittances, 277/999 Acknowledgments)
  • Implement and optimize Delta Lake tables with ACID transactions, Z-ordering, partitioning strategies, and data versioning (time travel) for petabyte-scale healthcare data
  • Build MongoDB sync pipelines to deliver Gold-layer data into operational MongoDB collections for real-time application access
  • Develop/Maintain Scala-based reporting and workflow orchestration engines, including dynamic report generation, AP analysis, and notification triggers
  • Integrate with Azure ecosystem: Azure Blob Storage, Azure Data Lake, Azure AD, Azure DevOps CI/CD pipelines, and Databricks Unity Catalog
  • Manage multi-tenant data architectures with per-client configurations, secrets management (Databricks secrets scopes), and feature flag-driven rollouts
  • Build and maintain SQL Server integrations via JDBC for workflow metadata, feature flags, report configurations, and state management
  • Implement data quality frameworks with validation layers at each pipeline stage (patient info, diagnosis codes, CPT codes, facility mappings, rendering providers)
  • Write ad-hoc data correction, migration, and backfill scripts for production data reconciliation
  • Participate in code reviews, PR validation pipelines, and engineering best practices (commit/lint, conventional commits)


Required Skills & Qualifications:


  • Core: 6–7 years of data engineering experience with 3+ years on Databricks/Spark
  • Languages: Proficient in PySpark (primary) and Scala/Spark; Python 3.11+
  • Data Lake: Deep expertise in Delta Lake — medallion architecture, ACID transactions, schema evolution, Z-ordering, partition pruning
  • Cloud: Strong Azure experience: Blob Storage, Data Lake Gen2, Azure AD, Key Vault, Azure DevOps
  • Databases: Production experience with MongoDB (aggregation pipelines, indexing, bulk sync) and SQL Server (JDBC, stored procedures)
  • EDI/Healthcare: Familiarity with HIPAA EDI standards (X12 837, 835, 277, 999) or willingness to learn quickly
  • Orchestration: Databricks Workflows, notebook orchestration, parameterized jobs
  • DevOps: Azure Pipelines (YAML), CI/CD for Databricks artifacts, PR validation
  • Data Quality: Experience implementing validation frameworks, data reconciliation, and error tracking


Preferred Qualifications:


  • Experience in healthcare/RCM domain — claims lifecycle, payer/provider workflows, CPT/ICD-10 coding would be a plus
  • Hands-on with Databricks Unity Catalog for data governance
  • Exposure to AI/ML integration in data pipelines — LLM orchestration, model observability (Langfuse or similar)
  • Experience with feature flag systems for gradual data pipeline rollouts
  • Multi-tenant SaaS data architecture design
  • Performance tuning Spark jobs at scale (shuffle optimization, broadcast joins, adaptive query execution)
  • Tech Stack Summary
  • Azure Databricks · PySpark · Scala/Spark · Delta Lake · MongoDB · SQL Server · Azure Blob Storage · Azure DevOps · Unity Catalog · EDI X12 (837/835/277/999) · Langfuse · Python 3.11 · JDBC · Azure AD


What You'll Impact:


  • Process thousands of healthcare claims daily across multiple clients
  • Reduce claim processing time through pipeline optimization
  • Enable AI-powered medical coding that improves coder productivity
  • Deliver real-time operational dashboards for revenue cycle teams
  • Ensure HIPAA-compliant data processing at every layer

Skills

Apply

See also