Associate Director / Director Data Engineer

We are seeking an experienced Data Engineering Leader to drive multiple tracks and complex data initiatives. As the Associate Director of Data Engineering, you will be responsible for driving strategic alignment and cross-functional collaboration and overseeing the delivery of highly scalable data platforms and products. You will lead a dedicated team of data engineers, ensuring technical excellence, operational efficiency, and the successful execution of our data roadmap. This role requires up to 50% hands-on contribution.

Responsibilities:

• Team Leadership: Manage and lead a high-performing team of data engineers, coordinating cross-functional dependencies and fostering a culture of continuous learning and technical excellence.
• Strategic Execution: Drive and execute the technical roadmap for key data infrastructure, pipelines, and products, balancing short-term delivery with long-term strategic goals.
• Technical Excellence: Drive operational efficiency across teams by enforcing engineering standards and SDLC best practices. Own the overall performance, reliability, observability, and accuracy of data pipelines. Continuously optimise highly scalable, fault-tolerant data pipelines.
• Mentorship and Collaboration: Mentor senior engineers and act as the first point of contact for external stakeholders. Identify and resolve technical bottlenecks and resource constraints. Help to recruit and onboard data engineering talent to scale the team effectively.
• Troubleshooting and Incident Response: Proactively identify, diagnose, and resolve critical data pipeline and platform issues, minimising downtime and ensuring system stability.

Requirements:

• Experience: 10 years of proven experience in data engineering, with a strong track record of managing complex technical programs.
• Architecture and System Design: Deep understanding of scalable data systems with strong system design and architecture review skills.
• Cloud and Data Ecosystems: Expert-level proficiency in data architecture, the Databricks ecosystem (Delta Lake, Unity Catalogue, etc. ), and the AWS data ecosystem.
• Databricks and Spark Expertise: Deep, hands-on expertise in Apache Spark, PySpark, and the Databricks ecosystem. Must have advanced knowledge of Delta Lake, Open Table Formats, Unity Catalogue, and Databricks Asset Bundles (DAB).
• Big Data and Architecture: Expert understanding of scalable data systems, Medallion Architecture, and distributed data pipelining. Strong system design and architecture review skills.
• AWS Data Ecosystem: High proficiency in AWS cloud ingestion and storage (S3), IAM, compute optimisations, and Infrastructure as Code (Terraform).
• Scaling and Optimisation: Proven experience in scaling data workloads, DataOps, compute optimisations (Docker, Kubernetes, Databricks cluster policies), and performance tuning (e. g., query optimisation, Z-ordering/Liquid clustering).
• AIML and Innovation: Familiarity with AI tools, MLOps, and techniques to enhance data engineering practice at scale and improve efficiency and quality of the pipelines.
• Data Domain Expertise: Prior knowledge of Finserv (especially credit, lending, or similar lines of business) and familiarity with regulatory data compliance is highly desirable.