Data Engineering Lead
Summary
Lead a team to build and maintain a large AWS-based data platform, designing scalable pipelines and setting engineering standards for batch and streaming data workflows.
We are recruiting a Data Engineering Lead to provide senior technical leadership across a large-scale AWS data platform supporting a major organisation.
This role combines hands-on technical expertise with team leadership, platform ownership and architectural decision-making. You will act as the senior authority on data engineering, helping ensure the platform remains reliable, scalable and capable of supporting increasingly complex data requirements.
You will work closely with senior data and architecture leaders while mentoring engineers and improving engineering standards across the wider organisation.
Key responsibilities
- Lead the development, operation and improvement of the AWS data platform
- Oversee the delivery of complex batch and streaming data pipelines
- Make technical decisions around architecture, data models, storage and engineering tooling
- Improve platform resilience, scalability, performance and availability
- Set standards across coding, testing, monitoring, documentation and DataOps
- Mentor and support data engineers at different levels
- Act as the senior escalation point for complex data engineering issues
- Work closely with Architecture, Information Security and Quality Assurance teams
- Collaborate with external partners on integrations and platform development
- Translate business requirements into practical technical solutions
- Help prioritise engineering work and support the organisation’s wider data strategy
- Ensure suitable operational controls and monitoring are in place
Technical environment
The platform uses a broad range of AWS services, including:
- Redshift
- S3
- Glue
- Lambda
- EC2
- Kinesis Firehose
The engineering environment also makes extensive use of Python, SQL and Bash, with data arriving through both legacy and modern source systems.
- Advanced SQL skills, including optimisation across very large datasets
- Experience building batch and streaming data pipelines
- Knowledge of data warehouses, data lakes and modern data storage patterns
- Experience with distributed processing technologies such as Spark
- Strong understanding of CI/CD, DevOps and DataOps practices
- Experience setting engineering standards and improving code quality
- Previous technical leadership or mentoring experience
- Strong stakeholder management and communication skills
- Experience working with exceptionally large datasets
- Exposure to complex operational environments
- Experience integrating legacy systems, fixed-width files and REST APIs
- Experience working with external technology and delivery partners