Senior Data Engineer (Databricks Migration)
Posted Updated 8
views
Summary
Build and migrate data pipelines using PySpark and Databricks, designing Lakehouse architectures and optimizing SQL/Spark workloads for a data platform transition.
- Analyze ETL ELT workloads
- Build PySpark data processing jobs
- Design Lakehouse architectures
- Develop and optimize data pipelines
- Implement Medallion Architecture layers
- Implement data governance with Unity Catalog
- Implement incremental processing
- Migrate data platform from BigQuery to Databricks
- Support production deployments
- Troubleshoot and optimize SQL and Spark workloads