Senior Data Engineer (Databricks Migration)
Posted Updated 8
views
Summary
Senior Data Engineer to migrate and modernize a data platform from BigQuery to Databricks, building PySpark pipelines and lakehouse architecture with Delta Lake and Unity Catalog.
- Analyze ETL ELT workloads and define migration approach
- Build PySpark data processing solutions
- Design lakehouse architecture with Databricks and Delta Lake
- Develop and optimize data pipelines for large volume datasets
- Implement data governance and security with Unity Catalog
- Implement incremental processing strategies
- Implement medallion architecture bronze/silver/gold
- Migrate data platform from BigQuery to Databricks
- Optimize SQL and Spark workloads
- Support production deployments and platform modernization