Databricks Data Engineer
Posted Updated
Description
- We are seeking a highly skilled Databricks Data Engineer to join our Data & AI practice.
- The successful candidate will have deep expertise in building scalable data pipelines, optimizing Lakehouse architectures and enabling advanced analytics and AI use cases on the Databricks platform.
- This role is critical in building and optimizing modern data ecosystems that enable data-driven decision making, advanced analytics, and AI capabilities for our clients
- As a trusted practitioner, you will design and implement robust ETL/ELT workflows, integrate real-time and batch data sources, and enable secure, well-governed data products and pipelines.
- You will thrive in a collaborative, client-facing environment, with a passion for solving complex data challenges, driving innovation and ensuring the seamless delivery of data solutions
- Primary Responsibilities;
- Client Engagement & Delivery
- Data Pipeline Development (Batch and Streaming)
- Databricks & Lakehouse Architectures
- Data Modelling & Optimisation (Delta Lake, Medallion architecture)
- Collaboration & Best Practices
- Quality, Governance & Security
- Business Relationships;
- Solution Architects
- Data Engineers, Developers, ML Engineers, and Analysts
- Client stakeholders up to Head of Data Engineering, Chief Data Architect, and Analytics leadership
- Measures of Success;
- Delivery of high-performing, scalable, and secure data pipelines aligned to client requirements
- High client satisfaction and successful adoption of Databricks-based solutions
- Demonstrated ability to innovate and improve data engineering practices
- Contribution to the growth of the practice through reusable assets, accelerators, and technical leadership
- Strong consulting values with ability to collaborate effectively in client-facing environments
- Strong problem-solving, analytical, and communication skills
- Hands-on expertise across the data lifecycle: ingestion, transformation, modelling, governance, and consumption
- Experience leading or mentoring teams of engineers to deliver high-quality scalable data solutions
- Proven experience in data engineering and pipeline development on Databricks and cloud-native platforms
- Understanding of data governance, security, and compliance frameworks
- Proficiency in ETL/ELT tools such as DBT, Matillion, Talend, or equivalent
- Hands-on experience with cloud platforms (AWS, Azure, GCP)
- Strong SQL and Python (or equivalent language) skills for data manipulation and automation
- Familiarity with Databricks Workflows and other orchestration tools
- Deep expertise with the Databricks platform (Spark/PySpark/Scala, Delta Lake, Unity Catalog, MLflow)
- Experience with version control tools (GitHub, Bitbucket) and CI/CD pipelines
- Exposure to AI/ML workloads desirable
- Familiarity with medallion architectures, data lakehouse principles and distributed data processing
- Knowledge of data modelling methodologies (star schemas, Data Vault, Kimball, Inmon)
- Experience: Minimum 5–8 years in data engineering, data warehousing, or data architecture roles, with at least 3+ years working with Databricks
- Preferred: BSc/MSc in Computer Science, Data Engineering, or related field
- Databricks certifications (Data Engineer Professional) highly desirable
- Education: University degree required
Skills
- AI
- Analytics
- Automation
- AWS
- Azure
- Bitbucket
- CI/CD
- Cloud
- Cloud Native
- Data Engineering
- Data Governance
- Data Modeling
- Data Pipelines
- Data Vault
- Data Warehousing
- Databricks
- dbt
- Delta Lake
- ELT
- ETL
- GCP
- GitHub
- Lakehouse
- Machine Learning
- Matillion
- Medallion Architecture
- MLflow
- PySpark
- Python
- Scala
- Spark
- SQL
- Talend
- Unity
- Unity Catalog
- Vault
- Version Control
Sign in to read and join the discussion.