Data Engineer
Posted
Summary
A data engineer role in Shoreditch building scalable ETL/ELT pipelines and cloud data platforms with Python, SQL, Databricks, Snowflake and Spark on AWS, Azure or GCP, owning solutions from design through production with data quality and observability in place.
The focus is on engineers who can build scalable data pipelines, develop cloud-based data platforms and deliver reliable data infrastructure to support analytics, machine learning and AI-driven applications.
I'm particularly interested in professionals who enjoy solving complex engineering problems, improving data reliability and taking ownership of technical solutions from design through to production.
Key responsibilities
- Design, build and maintain scalable ETL/ELT pipelines using Python and SQL.
- Develop and optimise data processing workflows across cloud-based environments.
- Build and maintain data platforms using technologies such as Databricks, Snowflake and cloud-native services.
- Work with large datasets using Apache Spark and distributed processing frameworks.
- Implement data quality checks, testing, monitoring and pipeline observability.
- Collaborate with analytics, software engineering, product and machine learning teams.
- Strong commercial experience with Python and SQL.
- Experience building and maintaining production data pipelines.
- Hands-on experience with AWS, Azure or GCP.
- Understanding of data modelling, warehousing and ETL/ELT architecture.
- Experience troubleshooting data quality, performance and reliability issues.
- Familiarity with Git, testing and software engineering best practices.