Data Engineer – Databricks (IBU Consumer)
Posted Updated 70
views
Summary
Builds and maintains scalable data pipelines on Databricks for consumer-facing IBU data products, developing ETL/ELT with PySpark, Spark SQL and Delta Lake, implementing medallion architecture, Unity Catalog and Workflows, and optimizing cluster performance, cost, and production job reliability.
Build and maintain scalable data pipelines on Databricks for consumer-facing IBU data products.
- Develop ETL/ELT pipelines using PySpark, Spark SQL and Delta Lake
- Implement medallion architecture (bronze/silver/gold), Unity Catalog and Workflows
- Optimize cluster performance and cost; monitor and troubleshoot jobs in production
Requirements
- Must have: 4+ yrs data engineering, Databricks, PySpark, SQL, Git/CI-CD.
- Nice to have: pharma/consumer data, Databricks certification