Senior Data Engineer (Databricks Migration)
Posted Updated 8
views
Summary
Senior Data Engineer to migrate and optimize analytics workloads from BigQuery to Databricks, building PySpark pipelines, Lakehouse architecture, and Delta Lake layers for large-scale data processing.
- Analyze ETL ELT workloads and define migration approach
- Build and optimize Spark data processing solutions with PySpark
- Collaborate on architecture discussions and technical solution design
- Conduct code reviews and improve platform reliability
- Design and implement Lakehouse architecture with Delta Lake
- Design and maintain medallion Bronze Silver Gold layers
- Develop and optimize data pipelines for large volume analytics
- Implement data governance and security with Unity Catalog
- Implement incremental processing strategies
- Migrate analytical platform from BigQuery to Databricks
- Support production deployments and platform modernization
- Troubleshoot and optimize SQL and Spark workloads