Senior Data Engineer - Scalable Data Pipelines & ML Ready
Posted Updated
Description
DFP Recruitment is seeking a Senior Data Engineer to design, build and maintain scalable data pipelines and data products to support data science, ML, and analytics. You will work with data scientists, platform teams and stakeholders to translate requirements into production-ready datasets and services, applying cloud, DevOps and data governance practices.
The role focuses on AWS-based data engineering, performance, security and observability, with opportunities to influence architecture and
Design, build, test and maintain scalable batch and event-driven data pipelines for analytics, machine learning and AI use cases. Develop curated, reusable and well-documented datasets and data products from structured and unstructured sources. Work with data scientists and machine learning engineers to productionise data preparation, feature engineering, model input and model output pipelines. Translate business and analytical requirements into technical designs, delivery estimates, work items and acceptance criteria. Implement automated data quality controls, validation, metadata, lineage, logging, monitoring and alerting. Optimise SQL, Python and cloud workloads for performance, reliability, security and cost efficiency. Apply infrastructure-as-code, version control, automated testing, CI/CD and environment management practices to data engineering solutions. Prepare and maintain technical documentation, including data mappings, interface specifications, runbooks, support procedures and architectural decisions. Collaborate with Agency data custodians, cyber security, privacy, platform, architecture and delivery teams to ensure solutions meet governance and operational requirements. Demonstrated experience designing and delivering production-quality data pipelines and data products in AWS, using services such as S3, Glue, Lambda, Athena, EMR, Kinesis, Step Functions or equivalent managed services. Strong experience with ETL/ELT patterns, data modelling, orchestration, data quality testing, metadata and lineage across structured and unstructured data. Demonstrated application of modern software engineering and DevOps practices, including Git, code review, automated testing, CI/CD and infrastructure-as-code using Terraform or AWS CloudFormation. Strong programming capability in Python, with experience developing maintainable, modular and testable production code. Experience with Java or Scala is also relevant. Advanced SQL skills, including query optimisation, performance tuning and working with large or complex datasets. Hands-on experience delivering cloud-based data solutions in containerised environments such as Docker, ECS or EKS, with a sound understanding of security, access controls, secrets management and operational monitoring.Skills
- Acceptance Criteria
- AI
- Analytics
- Athena
- AWS
- CI/CD
- Cloud
- CloudFormation
- Cybersecurity
- Data Engineering
- Data Governance
- Data Modeling
- Data Pipelines
- Data Quality
- Data Science
- DevOps
- Docker
- ECS
- EKS
- ELT
- ETL
- Event Driven Architecture
- Feature Engineering
- Git
- Infrastructure as Code
- Java
- Kinesis
- Lambda
- Machine Learning
- Observability
- Python
- S3
- Scala
- Secrets Management
- SQL
- Terraform
- Test Automation
- Version Control
Sign in to read and join the discussion.