Senior Data Engineer
About the role
A data engineering role focused on building, developing, and maintaining dbt transformation models across multiple data pipeline layers, from data ingestion to consumer-ready datasets. The role involves developing and optimising Snowflake SQL transformations, designing Apache Airflow DAGs for orchestrating data pipelines on AWS, and developing AWS Lambda-based data ingestion solutions using Python.
Key responsibilities
Build, develop, and maintain dbt transformation models across multiple data pipeline layers, from data ingestion to consumer-ready datasets.
Develop and optimise Snowflake SQL transformations to ensure data quality, accuracy, and performance.
Design, develop, and maintain Apache Airflow DAGs for orchestrating data pipelines on AWS.
Develop and maintain AWS Lambda-based data ingestion solutions using Python.
Create, maintain, and update Source-to-Target Mapping (STM) documentation, ensuring clear traceability between source systems and target datasets.
Monitor, troubleshoot, and optimise data pipelines to ensure reliability, scalability, and timely data delivery.
Collaborate with data engineers, analysts, and business stakeholders to understand data requirements and deliver fit-for-purpose datasets.
About you
Strong hands-on experience with Snowflake, dbt, Apache Airflow, and data modelling.
Strong SQL proficiency, including complex queries, data transformations, and query optimisation.
Hands-on experience developing and maintaining dbt models and data transformation workflows.
Experience developing and managing DAGs for data pipeline orchestration.
Working knowledge of Amazon S3, AWS Lambda, Amazon ECS, AWS Glue Data Catalog, and Amazon EventBridge.
Experience working with fixed-format financial data files.
Experience implementing and maintaining CI/CD pipelines using GitHub Actions.
Knowledge or experience with Apache Iceberg is desirable.