Lead Data Engineer - Databricks & Spark Platform
Posted Updated
Description
EPAM Systems is seeking a Lead Data Engineer to shape scalable ingestion, transformation, and governed data products on Databricks. You will balance hands-on delivery with technical leadership and architecture to enable self-service analytics and AI-driven engineering.
You will design ingestion/transformation frameworks, enforce Databricks best practices, and build end-to-end data products while mentoring engineers and driving reusable frameworks across teams.
Design scalable data ingestion and transformation frameworks on Databricks. Define and enforce Databricks best practices for medallion layers and operational excellence. Build batch and streaming pipelines using Apache Spark and PySpark. Deliver end-to-end data products from ingestion through governed consumption layers. Develop trusted analytical datasets and semantic models for reporting and self-service analytics. Implement data quality, lineage, monitoring, and observability standards. Lead proofs-of-concepts and evaluate emerging capabilities in the Databricks ecosystem. Mentor engineers and drive reusable frameworks adopted across teams. 5+ years of data engineering experience with production-grade platforms. Strong Databricks experience in enterprise production environments. Lead-level technical leadership to drive standards and mentor engineers. End-to-end delivery across ingestion, transformation, semantic layers, and reporting. Advanced Apache Spark and PySpark expertise for scalable pipelines. Advanced SQL skills for transformation, modeling and performance tuning. Upper-Intermediate English (B2) for leading technical discussions. Healthcare benefits Paid time off Upskilling & certification LinkedIn Learning access Global career opportunitiesSkills
Sign in to read and join the discussion.