Data Engineer with Databricks experience
Posted Updated
Summary
Data engineer (8+ years) designing and maintaining scalable data pipelines and Lakehouse architectures in Databricks for an organisation based in London/Woking (hybrid, 3 days in office), up to £80,000. Core stack: Databricks, SQL, Python, Apache Spark, Delta Live Tables, Unity Catalog, Azure DevOps.
Role – Data Engineer
Location – London/ Woking - (3 days in office)
Salary: Upto £80,000
We’re working with a forward-thinking organisation looking to bring in a Data Engineer to shape and drive their data strategy, building a scalable, business-critical analytics layer.
Key Responsibilities
- Should have atleast 8+ yrs of experienc in design, build, and maintain scalable, high-performance data pipelines and modern Lakehouse architectures using Databricks, ensuring reliable data integration, orchestration, and operational excellence.
- Develop robust, business-focused data models and warehouse solutions that support analytics, reporting, and enterprise data initiatives, leveraging dimensional modeling, Star Schema, SCD, CDC, and Medallion Architecture best practices.
- Engineer efficient ETL/ELT workflows using SQL, Python, Apache Spark, and Databricks technologies, including Delta Live Tables, Workflows, Unity Catalog, DataFrames, UDFs, and Delta Sharing.
- Optimize data platforms through proactive monitoring, root cause analysis, performance tuning, and issue resolution while ensuring data quality, reliability, and timely stakeholder communication.
- Leverage modern Databricks capabilities such as Lakeflow Connect, Lakebase, Metric Views, Genie, and other emerging platform features to drive innovation and continuous platform improvement.
- Collaborate closely with cross-functional teams including Business Analysts, Data Architects, BI Developers, and Product Owners to deliver scalable, secure, and fit-for-purpose data solutions aligned with business priorities.
- Implement DevOps best practices by utilizing Azure DevOps, Git repositories, CI/CD pipelines, and Agile/Scrum methodologies to streamline development, testing, and deployment processes.
- Contribute to engineering excellence by producing comprehensive technical documentation, adopting AI-assisted development tools (e.g., Cursor), and promoting coding standards, knowledge sharing, and continuous improvement across the data engineering team.
Preferred Qualifications
- Strong hands-on experience with Databricks, SQL, Python, Apache Spark, and Pandas.
- Proven experience designing and delivering enterprise data pipelines, data warehouses, lakehouses, and analytics platforms.
- Master's degree in Data Science (or equivalent) is desirable.
- Azure certifications (AZ-900, DP-900, DP-203, DP-500) and Databricks certifications (Lakehouse Data Engineer Associate/Professional) are advantageous.