Lead Data Engineer
Summary
Lead Data Engineer in the UK Government's NDQIS, designing and delivering end-to-end data pipelines, ETL and automation for policing data quality. Day to day involves building Python/AWS-based pipelines, ensuring data governance and security, managing a team, and collaborating with stakeholders, on a hybrid basis from Salford (60% office).
As Lead Data Engineer within the NDQIS, you design and deliver end-to-end data solutions that support data-driven decision making for policing data quality. You’ll translate complex requirements into scalable, secure data pipelines while upholding governance and security standards. You’ll drive data integration, ETL and automation, collaborating with multiple teams to align with organisational priorities. Your work shapes data engineering strategy, boosting reliability, maintainability and cost efficiency. This role offers a visible impact on national data quality initiatives and cross-functional collaboration.
Pay / Benefits- Civil Service Pension with employer contribution
- 25 days annual leave plus 8 public holidays and extra privilege day
- Hybrid working options with 60% office presence
- In-year reward scheme for exceptional achievements
- Non-contractual capability-based allowance
- Competitive base salary with London range
- Manage data sources to improve quality and readiness for analysis, balancing availability, capacity and archiving; continually improve data ingestion processes
- Develop, maintain and support end-to-end data solutions including pipelines, ETL, data profiling and documentation of large-scale datasets
- Lead stakeholder engagement to manage expectations, translate needs, and foster a service-oriented data culture
- Ensure data solutions comply with data privacy and information security requirements; stay current with industry and government best practices
- Oversee supplier performance and data engineering team capability; roadmap development and tooling modernization including data analytics re-platforming
- Line management, coaching and professional development of the team
- Collaborate with Principal Data Engineer, Data Architects, DBAs, Solution Architects and Analytics team to set data strategy and governance
- Deliver complex data flows connecting operational systems, analytics sources and BI platforms
- Leading and delivering data engineering projects and teams
- Translating data quality rules into automated, testable pipeline logic using Python, regex and NLP
- Proficiency in IT programming and automation tools (Git, GitHub, CI/CD)
- Experience with Cloud Data technologies (AWS: EC2, VPC, RDS, IAM, Lambda, CloudWatch, Docker)
- Experience with ETL tools, databases and orchestration tools; SQL, Linux, Bash; PostgreSQL; AWS Glue, Step Functions
- Familiarity with data visualization tools (Power BI) and emergent AI/ML/data automation contexts
- Ability to gather requirements, present findings to technical and non-technical audiences and senior stakeholders
- Knowledge of data privacy and information security requirements
- Stakeholder management
- Clear communication to both technical and non-technical audiences
- Collaborative mindset and cross-team coordination
- Python and related libraries; regex; NLP
- API integration; JSON/CSV handling
- Git, GitHub; CI/CD pipelines