Lead Data Engineer
Summary
Lead a data engineering team at an e-commerce platform in East London (hybrid), architecting scalable ETL/ELT pipelines, cloud data warehouses and lakes, and data governance. Core stack includes Spark, Kafka, SQL, Python/Scala, and AWS/Azure/GCP cloud data services.
About the Role
Our client, a prominent e-commerce platform, is seeking an experienced Lead Data Engineer to join their growing data team in East London. This role is responsible for designing, building, and maintaining scalable data pipelines and infrastructure that support the company's analytics and machine learning initiatives. You will lead a team of data engineers, architect data solutions, and ensure the efficiency and reliability of the data platform. The ideal candidate has a strong background in data warehousing, ETL processes, big data technologies, and cloud data services. This position offers a hybrid work arrangement, combining remote flexibility with essential in-office collaboration in East London.
Key Responsibilities
- Design, develop, and maintain robust and scalable ETL/ELT pipelines to ingest and process large volumes of data.
- Architect and build data warehouses and data lakes on cloud platforms (e.g., AWS , Azure , GCP ).
- Optimize data infrastructure for performance, scalability, and cost-efficiency.
- Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver solutions.
- Implement data governance and data quality best practices.
- Lead and mentor a team of data engineers, providing technical guidance and support.
- Monitor and troubleshoot data pipelines and infrastructure issues.
- Evaluate and implement new data technologies and tools.
- Ensure data security and compliance with relevant regulations.
- Automate data processes and workflows using scripting and programming languages.
Requirements
- Bachelor's degree in Computer Science, Engineering, or a related quantitative field; Master's degree is a plus.
- 5+ years of experience in data engineering, with at least 2 years in a lead or senior role.
- Proven experience designing and building data pipelines using tools like Apache Spark , Kafka , or similar.
- Strong experience with cloud data services (e.g., AWS Redshift, S3, Glue; Azure Data Factory, SQL Data Warehouse; Google BigQuery, Cloud Storage).
- Proficiency in SQL and experience with relational and NoSQL databases.
- Experience with programming languages such as Python or Scala.
- Familiarity with data warehousing concepts and dimensional modeling.
- Excellent problem-solving, analytical, and communication skills.
- Demonstrated leadership and team management capabilities.
- Experience working in an e-commerce environment is a significant advantage.
- Ability to work a hybrid schedule from our offices in East London.
Benefits
Our client offers a competitive salary, comprehensive health and wellness benefits, performance bonuses, and opportunities for continuous learning and professional development. This hybrid role provides flexibility while fostering team collaboration. Join a data-driven organization in East London that values innovation and offers a challenging and rewarding career path in the exciting field of big data.