Lead/Principle Data Engineer
Summary
Lead/Principal Data Engineer designing, building and optimising secure batch and real-time data pipelines for large-scale government data platforms in an Agile, cross-functional team. Core stack is AWS (EC2, RDS, Lambda, Athena, Glue), Python, relational/NoSQL databases, with TDD and strong data modelling.
You will design, build, and optimise secure data pipelines for large-scale government data platforms. In an Agile, cross-functional team, you’ll deliver batch and real-time processing solutions that support critical decision-making. This role emphasises secure environments, modern cloud tech, and open-source tools to drive reliable data delivery and platform evolution. You’ll work closely with engineers and stakeholders to ensure data quality, performance, and compliance, contributing to mission-critical outcomes.
Responsibilities- Develop, test, and maintain robust data pipelines meeting business requirements
- Design and optimise batch and real-time data processing solutions
- Contribute to logical/physical database design across data platforms
- Enhance platform capabilities to tackle new data challenges
- Analyse complex datasets to improve processing, quality, and performance
- Ensure data models align with architecture standards and definitions
- Perform root cause analysis on data issues and implement solutions
- Collaborate with engineers, architects, and stakeholders to deliver scalable data solutions
- Support continuous improvement and best practices in data engineering and platform development
- Produce technical documentation for pipelines and system enhancements
- Experience building and maintaining data pipelines with modern data processing frameworks
- Experience with Agile delivery in fast-paced environments
- Strong data modelling knowledge and understanding of data structures
- Experience with AWS (EC2, RDS, Lambda, Athena, Glue) applications
- Relational and NoSQL database experience with query tuning and optimization
- Proficiency in Python
- Experience applying Test-Driven Development (TDD)
- Strong analytical and problem-solving abilities
- Excellent communication and cross-functional collaboration
- Bachelor’s degree or equivalent experience
- Collaborative mindset
- Strong communication
- Analytical thinking
- AWS services (EC2, RDS, Lambda, Athena, Glue)
- Data pipelines and processing frameworks
- Data modelling and database design