Data Engineer
Summary
Builds and maintains scalable data pipelines using PySpark and AWS tools to process large datasets, enabling analytics and business intelligence for UK government programs.
Salary: £50,000 - 50,000 per year
Requirements:- Experience developing and managing PySpark workflows within AWS environments.
- Knowledge of AWS Lambda and event-driven solutions to support automated data processing.
- Experience provisioning and supporting cloud infrastructure using Amazon EC2 and automation tools.
- Understanding of Agile ways of working and experience contributing to CI/CD delivery practices.
- Ability to optimise data processing, storage, and queries to improve performance and data quality.
- Experience designing and automating ETL pipelines using PySpark and Spark SQL.
- Strong full stack development background with technologies including .NET 4.8, C#, JavaScript, C++, and Java.
- Help deliver modern data engineering solutions that support analytical reporting, business intelligence, and strategic decision-making.
- Design, build, maintain, and optimise scalable data products used by analysts and data scientists.
- Develop and orchestrate PySpark-based ETL pipelines using Amazon EMR, AWS Lambda, and EC2.
- Process large volumes of structured and unstructured data efficiently in a hybrid cloud environment.
- Collaborate with stakeholders to translate reporting requirements into robust technical solutions.
- Maintain high standards of data quality, performance, and security.
- Contribute to Agile delivery practices and support CI/CD automation.
- Maintain technical documentation, conduct code reviews, and share knowledge with other engineers and data professionals.
- AWS
- Lambda
- Business Intelligence
- C#
- CI/CD
- Cloud
- EC2
- ETL
- Support
- Java
- JavaScript
- PySpark
- SQL
- Security
- Spark
- ASP.NET
More:
We are DWP Digital, part of the UKs largest government department, and we use fresh ideas and leading-edge technology to build and maintain digital solutions used by nearly every person in the UK at key moments in their lives. Our Management Information team works in a hybrid model, spending 40% of time working from home and 60% collaborating face to face in a hub. We offer competitive pay of up to £51,528, a generous civil service pension with employer contributions of 28.97%, and a leave package starting at 26 days rising to 31 days, plus up to 3 extra flexi days each month. Our benefits also include volunteering and charitable giving time, discounts and savings, interest-free loans for bikes or season tickets, and professional development, coaching, mentoring, and career progression opportunities. We have an award-winning, inclusive culture and have been recognised for diversity, leadership, and being a great place to work in digital.
last updated 34 week of 2026