Senior Platform Data Engineer
Summary
The Senior Platform Data Engineer will manage clinical data pipelines and RAG infrastructure using Databricks and vector databases to support AI model training. The role focuses on streaming healthcare data, maintaining feature stores, and ensuring robust data governance within a clinical environment.
Responsibilities
- Stream data from Epic SDE, ADT feeds, and clinical sources into Databricks
- Curate shared clinical feature tables in Databricks/Unity Catalog for AI model training and monitoring
- Own and operate RAG (Retrieval-Augmented Generation) infrastructure, including ingestion, chunking, and embedding pipelines
- Administer vector databases, including schema design, indexing, and performance tuning
- Build and maintain retrieval pipelines using hybrid search and reranking
- Manage Databricks workspace configuration, Unity Catalog governance, and cost monitoring
- Administer the Feature Store and manage user access controls
Requirements
- 5+ years of experience in data engineering with batch and streaming pipelines
- Expert proficiency with Databricks (Delta Live Tables, PySpark, Unity Catalog, Feature Store)
- Hands-on experience with real-time data ingestion frameworks like Kafka or Spark Structured Streaming
- Strong SQL and Python (pandas, PySpark) skills Experience administering Databricks workspaces and compute management
- Understanding of data governance, lineage, and quality monitoring
- Bachelor's degree in a related field; Master's degree preferred
Preferred Qualifications
- Familiarity with clinical data models and healthcare data sources (EHR, ADT, claims)
- Experience with Epic data extraction methods (SDE, FHIR, epic-ws)
Benefits
- Healthcare benefits including vision and dental from day one
- Benefits for domestic partners
About the Company
Geisinger is a healthcare system serving over 1 million people, comprising nine hospital campuses, a health plan, two research centers, and the Geisinger Commonwealth School of Medicine.