Data Engineer (Data lake/Data warehouse)
Summary
Designs and maintains cloud-based data pipelines and lakehouse architectures for a weather data platform, enabling analytics and AI/ML workloads.
Our client is seeking a Senior Data Engineer to support the development and enhancement of a large-scale Weather Data Lakehouse platform. The platform serves as a central repository for meteorological, climate, and operational data, enabling analytics, AI/ML initiatives, research, and operational forecasting.
Responsibilities
- Design, develop, and maintain scalable data ingestion, transformation, and processing pipelines.
- Build and enhance cloud-based data lakehouse architectures for analytics and AI/ML workloads.
- Implement data engineering best practices including data governance, metadata management, monitoring, and automation.
- Optimize data platform performance, scalability, reliability, and operational efficiency.
- Support CI/CD, DevOps, and SRE practices to improve deployment and system resiliency.
- Troubleshoot and resolve complex data pipeline and platform issues.
- Ensure compliance with security requirements and enterprise architecture standards.
- Collaborate with cross-functional teams to deliver high-quality data solutions.
Requirements
- Degree in Computer Science, Data Engineering, Information Systems, or related disciplines.
- Strong experience in designing and implementing enterprise-scale data platforms, data lakes, or lakehouse solutions.
- Hands-on experience in data engineering, ETL/ELT development, cloud technologies, and distributed data processing frameworks.
- Experience with data pipeline orchestration, automation, monitoring, and CI/CD practices.
- Familiarity with DevOps and SRE principles in production environments.
- Strong analytical, problem-solving, and stakeholder management skills.