Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

Shift

Lead Data Platform Engineer — Azure Databricks, Hybrid

Posted Updated

Description

Shift is seeking an experienced Data Engineer to own the Azure Databricks platform, set standards and roadmap, and lead the data workstream for a core lending system migration. You’ll be the senior engineer in a small team, with two junior engineers, and ample room to shape how data works at Shift.

You will design robust ELT pipelines, build a scalable dimensional data model, lift data quality and cost efficiency, and mentor teammates while collaborating with analysts, data scientists and

Own the data platform roadmap and engineering standards, including CI/CD, separate environments, automated testing, monitoring and access governance through Unity Catalog. Design and lead the build of robust, scalable ELT pipelines on Azure Databricks. Lead the data workstream for our core lending platform migration: remap pipelines and models, and reconcile outputs before and after cutover alongside Engineering. Build and extend a curated, dimensional data model that becomes the default place the business goes for trusted, self-serve data. Lift data quality, freshness and alerting so issues are caught before the business notices them. Keep the platform cost-efficient through cluster optimisation, job orchestration and smart use of compute. Mentor and develop our junior data engineers through code review, pairing and clear standards. Partner with analysts, data scientists, product managers and Engineering to turn business needs into data products people trust. 5+ years in data engineering with hands-on Azure Databricks experience. Strong Python and SQL, plus CI/CD and deploying pipelines as code (Azure DevOps or GitHub, Databricks Asset Bundles). A track record of lifting standards on an existing platform, not just building something new on a clean slate. Solid dimensional modelling experience (star schemas, conformed dimensions, medallion architecture). Experience mentoring engineers and leading technical decisions through design and code review. The ability to explain technical direction clearly to non-technical stakeholders, and an instinct to work across teams rather than in a silo. Experience leading the data side of a core system migration is a strong plus. Hybrid work policy Parental leave up to 26 weeks Strategy Week Flat structure Wellbeing benefits

Skills

What Lead Data Engineering jobs ask for — and how much of it you have →
Apply

See also

Data Engineering jobs by country — openings, pay and top skills →