Data Engineer
Posted 3
views
Summary
Contract data engineer (initial 6 months, inside IR35) designing and optimising large-scale ETL/ELT pipelines on Databricks using PySpark, Spark SQL and Delta Lake medallion architectures on Azure/AWS, with governance via Unity Catalogue and CI/CD practices. Predominantly remote with some travel to a London client site; SC clearance eligibility required.
Salary: £29,000 - 69,000 per year Requirements: Strong commercial experience with Databricks Excellent PySpark, Apache Spark and Spark SQL skills Experience building production-grade ETL/ELT pipelines Strong knowledge of Delta Lake and Lakehouse architecture Experience implementing Medallion/Bronze-Silver-Gold architectures Strong experience with Azure and/or AWS data platforms Knowledge of Unity Catalogue, data governance, data quality and security Experience integrating Databricks with wider …