Data Engineer
Posted Updated 1
view
Summary
A 15-month contract (inside IR35) Data Engineer role, hybrid with 2-3 days per week onsite in Cambridge, UK. The contractor builds scalable batch and streaming pipelines for a petabyte-scale AWS lakehouse using Databricks, Spark, Kafka and Python/Scala, with emphasis on DataOps, data governance, and mentoring.
Whitehall Resources are looking for a Data Engineer. This role is hybrid working with 2-3 days per week onsite in Cambridge, and the remainder remote working, for an initial 15 month contract.
***Inside IR35***
Responsibilities:
- Build highly scalable, fault-tolerant distributed data processing systems (batch and streaming) that handle tens of terabytes of data daily, supporting a petabyte-scale data lakehouse.
- Collaborate with Engineering stakeholders to own and deliver intelligence data products. Negotiate delivery milestones and build new data flows to ensure performance, scalability, and cost efficiency.
- Participate in architectural discussions, influence the product roadmap, and lead new initiatives from concept to delivery.
- Data quality, governance, compliance and observability across all stages of the data lifecycle.
- Serve as a mentor and data advocate, providing technical leadership and supporting team members within the business unit and organisation.
Required Skills and Experience:
- Degree and/or equivalent experience in engineering, computer science or a related data-intensive field.
- Proficient with one or more programming languages (Python, Scala).
- Hands-on experience with modern lakehouse technologies (e.g. Databricks, Spark, Kafka).
- Comfortable building pragmatic ETL/ELT workflows in AWS using orchestration frameworks or cloud-native tools.
- DataOps mentality. Understanding of how to apply automation, version control, testing, monitoring and continuous delivery to data systems.
- Excellent written and verbal communication skills, with the ability to collaborate effectively in fast-paced, diverse environments.
- Ability to present data and insights in an engaging and clear way, with strong attention to detail, and ensure data accuracy and quality.
- Solid understanding of data governance, security, and compliance frameworks.
- Experience with data visualisation (Tableau, Looker, Kibana or Power BI).
- Experience usingAI/ML data pipelines, MLOps, or related workflows and the processes around testing, monitoring, and SLAs.
- Passion for mentoring other engineers and contributing to the development of engineering standards.