Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

Information Tech Consultants

Big Data Developer

Posted Updated 39 views

Summary

A data engineer role: design, build, and maintain scalable batch and real-time data pipelines and ETL/ELT workflows supporting analytics and BI. Core stack is SQL, Python, Apache Spark, Airflow, and a cloud platform (AWS/Azure/GCP), with work across data warehouses and data lakes. London-based, UK candidates only.

Education: Bachelors in Science (IT/Computer Science/Engineer)

Employment Type: Full-Time

UK based candidates only.

Job Summary

We are looking for a skilled Data Engineer to design, develop, and maintain scalable data pipelines and data infrastructure. The ideal candidate will have experience in data integration, ETL/ELT processes, data warehousing, and cloud-based data platforms to support analytics and business intelligence.

Key Responsibilities

  • Design, develop, and maintain robust data pipelines for batch and real-time data processing.
  • Build and optimize ETL/ELT workflows to extract, transform, and load data from multiple sources.
  • Develop complex SQL queries and optimize database performance.
  • Work with data warehouses, data lakes, and distributed data processing frameworks.
  • Implement data quality checks, validation, monitoring, and error-handling mechanisms.
  • Collaborate with data analysts, data scientists, BI developers, and business stakeholders.
  • Ensure data security, governance, integrity, and compliance with organizational standards.
  • Troubleshoot pipeline failures and improve data processing performance.
  • Maintain technical documentation for data models, pipelines, and workflows.

Required Skills and Qualifications

  • Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
  • 2-5 years of experience in data engineering or a related role.
  • Strong proficiency in SQL and Python.
  • Experience with ETL/ELT tools and data integration frameworks.
  • Knowledge of relational and NoSQL databases.
  • Experience with data warehousing concepts, dimensional modeling, and database optimization.
  • Familiarity with Apache Spark, Apache Airflow, or similar data processing and orchestration tools.
  • Experience with at least one cloud platform: AWS, Microsoft Azure, or Google Cloud Platform.
  • Understanding of data pipeline monitoring, testing, and performance optimization.
  • Strong analytical, problem-solving, and communication skills.

Preferred Qualifications

  • Experience with Snowflake, Databricks, Amazon Redshift, Google BigQuery, or Azure Synapse Analytics.
  • Knowledge of Kafka or other streaming technologies.
  • Familiarity with Docker, Kubernetes, Git, and CI/CD pipelines.
  • Experience with data lakehouse architecture and cloud-native data solutions.
  • Knowledge of data governance, metadata management, and data cataloging.

Skills

Apply

See also