AI-Driven Databricks Data Engineer
Posted Updated
Description
Join Deloitte NZ as a Databricks Data Engineer to design, build, and maintain scalable data and AI solutions on Databricks. You'll work across ingestion, ELT/ETL, modelling, orchestration, deployment, and production support, leveraging Lakebase, Unity Catalog, and Databricks Apps.
Your AI-enabled engineering mindset will help you drive GenAI-powered development tools to accelerate delivery while upholding security and quality standards.
Design, develop, and maintain scalable data and AI solutions on Databricks. Build robust data ingestion, ELT/ETL, and data processing pipelines using Databricks. Develop high-quality and reusable transformations through Lakeflow Declarative Pipelines. Design and implement Lakehouse architectures. Build dimensional, curated, and feature-ready data models to support reporting, analytics, machine learning, and Generative AI use cases. Develop and maintain integrations between Databricks and cloud platforms, APIs, databases, files, streaming platforms, and enterprise applications. Implement automated CI/CD pipelines and modern DevOps practices for Databricks deployments. Develop and orchestrate workloads using Lakeflow Jobs, Azure Data Factory, Airflow, or similar technologies. Implement appropriate security, governance, data quality, lineage, and monitoring controls. Optimise Databricks workloads for performance, scalability, reliability, and cost efficiency. Troubleshoot complex platform, pipeline issues across development and production environments. Collaborate with architects, analysts, data scientists, ML engineers, and business stakeholders to translate requirements into practical technical solutions. Contribute to engineering standards, reusable frameworks, technical documentation, and continuous improvement initiatives. Proactively leverage Genie Code, GitHub Copilot, and other AI-powered development tools across solution design, coding, testing, troubleshooting, code review, and documentation. Identify opportunities to use AI and automation to reduce repetitive engineering effort and accelerate the development lifecycle. Strong hands-on experience with Databricks. Experience designing and building Lakehouse architectures. Strong understanding of data engineering, data modelling, data warehousing, and distributed processing concepts. Experience building batch and/or streaming data pipelines. Experience with orchestration technologies such as Lakeflow Jobs, Azure Data Factory, Airflow, or similar tools. Understanding of cloud data services across Azure and/or AWS. Experience with Git, CI/CD, and modern software engineering practices. Experience using Python in data engineering solutions are advantageous. Familiarity with infrastructure-as-code technologies such as Terraform or similar. Strong analytical, troubleshooting, and problem-solving skills. Excellent communication and stakeholder engagement skills. Flexible work Qualifications subsidies Health insurance Income protection 22 days leave Social events Well-being focus SustainabilitySkills
- AI
- Airflow
- Analytics
- API
- Automation
- AWS
- Azure
- Azure Data Factory
- CI/CD
- Cloud
- Data Engineering
- Data Ingestion
- Data Modeling
- Data Pipelines
- Data Quality
- Data Warehousing
- Databricks
- DevOps
- ELT
- ETL
- Generative AI
- Git
- GitHub
- Github Copilot
- Infrastructure as Code
- Lakehouse
- Machine Learning
- Python
- Solution Design
- Terraform
- Unity
- Unity Catalog
Sign in to read and join the discussion.