Senior Data Engineer: Azure Databricks & PySpark
Posted Updated
Description
EPAM Systems seeks a Senior Data Engineer to design and deliver scalable cloud-native data solutions on Azure and Databricks. You will craft ETL/ELT pipelines with PySpark, Spark SQL, and Delta Lake, while enforcing data governance and robust quality controls.
You will collaborate with stakeholders to capture requirements, optimize workloads, and drive best practices in CI/CD and infrastructure automation.
Design, build, and sustain scalable data ingestion, transformation, and delivery pipelines. Deliver cloud-native data solutions using Azure and Databricks technologies. Develop ETL/ELT processes with PySpark, Spark SQL, and Delta Lake. Apply metadata-driven ingestion frameworks to boost scalability and maintainability. Build batch and incremental data processing pipelines and ensure data quality. Set up monitoring, logging, and error handling mechanisms; address production issues. Track and optimize Databricks workloads for efficiency and cost control. Guide team members and shape engineering standards and best practices. 3+ years of experience in Data Engineering or Software Engineering. Expertise in Azure Databricks, Apache Spark, and PySpark. Proficiency in Spark SQL with cloud-based Azure solutions. Advanced Python programming and SQL optimization. Knowledge of Unity Catalog and data governance practices. Familiarity with Git-based workflows and CI/CD pipelines. Experience with Databricks Workflows and Azure Data Factory. Strong communication and stakeholder management skills. Healthcare benefits Paid time off Upskilling and certification courses LinkedIn Learning accessSkills
Sign in to read and join the discussion.