Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

DFP Recruitment

Senior Data Engineer - Scalable Data Pipelines & ML Ready

Posted Updated

Description

DFP Recruitment is seeking a Senior Data Engineer to design, build and maintain scalable data pipelines and data products to support data science, ML, and analytics. You will work with data scientists, platform teams and stakeholders to translate requirements into production-ready datasets and services, applying cloud, DevOps and data governance practices.

The role focuses on AWS-based data engineering, performance, security and observability, with opportunities to influence architecture and

Design, build, test and maintain scalable batch and event-driven data pipelines for analytics, machine learning and AI use cases. Develop curated, reusable and well-documented datasets and data products from structured and unstructured sources. Work with data scientists and machine learning engineers to productionise data preparation, feature engineering, model input and model output pipelines. Translate business and analytical requirements into technical designs, delivery estimates, work items and acceptance criteria. Implement automated data quality controls, validation, metadata, lineage, logging, monitoring and alerting. Optimise SQL, Python and cloud workloads for performance, reliability, security and cost efficiency. Apply infrastructure-as-code, version control, automated testing, CI/CD and environment management practices to data engineering solutions. Prepare and maintain technical documentation, including data mappings, interface specifications, runbooks, support procedures and architectural decisions. Collaborate with Agency data custodians, cyber security, privacy, platform, architecture and delivery teams to ensure solutions meet governance and operational requirements. Demonstrated experience designing and delivering production-quality data pipelines and data products in AWS, using services such as S3, Glue, Lambda, Athena, EMR, Kinesis, Step Functions or equivalent managed services. Strong experience with ETL/ELT patterns, data modelling, orchestration, data quality testing, metadata and lineage across structured and unstructured data. Demonstrated application of modern software engineering and DevOps practices, including Git, code review, automated testing, CI/CD and infrastructure-as-code using Terraform or AWS CloudFormation. Strong programming capability in Python, with experience developing maintainable, modular and testable production code. Experience with Java or Scala is also relevant. Advanced SQL skills, including query optimisation, performance tuning and working with large or complex datasets. Hands-on experience delivering cloud-based data solutions in containerised environments such as Docker, ECS or EKS, with a sound understanding of security, access controls, secrets management and operational monitoring.

Skills

What Senior Data Engineering jobs ask for — and how much of it you have →
Apply

See also

Data Engineering jobs by country — openings, pay and top skills →