Data Engineer - Databricks
Posted Updated 7
views
Summary
Build and optimize Databricks-based lakehouse pipelines, including Delta Live Tables, ETL/ELT, streaming ingestion, and ML-ready datasets with MLflow and Unity Catalog.
- Build CI/CD pipelines for Databricks
- Build Delta Live Tables pipelines
- Configure Unity Catalog for governance
- Design ETL/ELT pipelines
- Document solutions and runbooks
- Implement medallion architecture
- Ingest streaming and incremental data
- Optimize lakehouse data platforms
- Orchestrate and monitor production workloads
- Prepare ML ready datasets with MLflow
- Query and profile datasets
- Set up PII masking
- Tune clusters for performance and cost