data engineer for the Net-Zero Data Public Utility
Summary
Builds and maintains scalable Python data pipelines for the Net-Zero Data Public Utility, transforming public-good climate data from APIs and structured files (Excel, Parquet) into clean, structured datasets using Pandas and Pydantic, documented and deployed to the cloud via CI/CD.
Описание:
The Climate Data Utility Tech Team develops scalable data pipelines for the Net-Zero Data Public Utility, which provides open and accessible public-good data through its website and APIs.
Задачи:
Design and develop scalable Python data pipelines to read from APIs and structured files such as Excel and Parquet Translate domain- and source-specific business logic into efficient code that transforms source data into structured data for downstream applications Implement data transformations and structuring with Pandas and Pydantic to ensure data quality, consistency, and compliance with business logic Collaborate with data scientists and analysts to support data-driven decision-making Document data engineering processes, including data lineage, data dictionaries, and system architectures Maintain best practices through code reviews, version control, and adherence to industry standards Deploy production-quality code through CI/CD pipelines into the cloud environment Mentor and guide junior team members in data engineering practices
Требования:
7+ Years of experience in data engineering or a similar role Proficiency in Python Experience building and managing data pipelines Knowledge of data warehousing and ETL processes Excellent problem-solving and communication skills
Условия:
Location: London, England, United Kingdom