Senior Data Engineer — AWS Data Pipelines & ML-Ready Data
Posted Updated
Description
DFP Recruitment is seeking a Senior Data Engineer to design, build and maintain reliable data pipelines and data products enabling data science, ML and AI initiatives. You will work with data scientists, solution architects and platform teams to translate complex data requirements into production-ready datasets and services.
You will apply software engineering, cloud and DevOps practices to ensure data quality, lineage, security and monitoring, and you'll optimize SQL, Python and cloud
Design, build, test and maintain scalable batch and event-driven data pipelines for analytics, machine learning and AI use cases. Develop curated, reusable and well-documented datasets and data products from structured and unstructured sources. Work with data scientists and machine learning engineers to productionise data preparation, feature engineering, model input and model output pipelines. Translate business and analytical requirements into technical designs, delivery estimates, work items and acceptance criteria. Implement automated data quality controls, validation, metadata, lineage, logging, monitoring and alerting. Optimise SQL, Python and cloud workloads for performance, reliability, security and cost efficiency. Apply infrastructure-as-code, version control, automated testing, CI/CD and environment management practices to data engineering solutions. Prepare and maintain technical documentation, including data mappings, interface specifications, runbooks, support procedures and architectural decisions. Collaborate with Agency data custodians, cyber security, privacy, platform, architecture and delivery teams to ensure solutions meet governance and operational requirements. Production-ready data pipelines and data products in AWS. Strong ETL/ELT, data modelling and data quality controls. Experience with Python and modern DevOps practices. Ability to translate business requirements into technical designs.Skills
Sign in to read and join the discussion.