Data Engineer (Ref 20994D)
Summary
Build and maintain scalable data pipelines on Azure Databricks, ensuring data quality and efficient ETL processes for migrating and deploying datasets.
Responsibilities:
- Creating data pipelines, ensuring data quality, and implementing ETL processes to migrate and deploy data across systems.
- To build and maintain scalable pipelines on Azure Databricks, with strong SQL and relational database expertise and solid DevOps/SDLC practices from design through deployment.
- Develop data connectors and pipelines to onboard agency datasets efficiently
Requirement :
- Degree in Computer Science with 3 to 5 years hands on experience in Databricks
- Hands-on experience building data pipelines and notebooks in Azure Databricks
- Strong SQL skills — complex queries, stored procedures, performance tuning
- Extensive experience working with relational databases (design, querying, optimization)
- Working knowledge of Delta Lake and Lakehouse architecture
- Experience with Azure Data Factory or similar orchestration tools
- Familiarity with Azure DevOps and CI/CD pipelines
- Solid understanding of data modeling concepts
- Strong analytical and troubleshooting skills for pipeline/query performance
- Comfortable leveraging Gen AI tools to boost day-to-day productivity
- Good communication skills for cross-functional collaboration
Shortlisted candidates will be offered a 1 Year agency contract employment
License No : 12C6060