Databricks Data Engineer
Summary
Build and optimize scalable data pipelines on Databricks using Spark and Delta Lake, implementing ETL/ELT processes and ensuring data quality for analytics and compliance.
Key Deliverables:
- Build and optimize scalable data pipelines using Apache Spark on Databricks
- Implement ETL/ELT processes with Databricks Notebooks and Delta Lake
- Ensure data quality, integrity, and compliance across platforms
- Support CI/CD, cloud migrations, and infrastructure automation
Role Responsibilities:
- Collaborate with data scientists and analysts to deliver data products
- Maintain and optimize large-scale distributed data jobs
- Use Azure DevOps, GitHub Actions, and Terraform for workflow automation
- Work with structured and unstructured data using Lakehouse architecture