Lead Data Engineer - PySpark & Azure (Insurance)
Posted
Summary
Lead a data engineering team improving an insurance investment reporting system: build ETL pipelines and cloud-native architecture on Azure (Data Factory, Databricks, Functions, Event Grid, Service Bus) using PySpark, implement CI/CD and infrastructure automation, and mentor engineers.
We are seeking a highly skilled Data Engineer with expertise in PySpark and Azure to lead the improvement of our Investment Reporting System.
This role involves guiding the engineering team, implementing cloud-native solutions, and ensuring adherence to best practices.
Key Responsibilities
- Lead the development of ETL pipelines and cloud-native architectures using Azure services.
- Work hands-on with Azure Data Factory, Databricks, Functions, Event Grid, and Service Bus.
- Collaborate with Solution Architects to translate designs into actionable tasks.
- Implement DevOps practices, CI/CD pipelines, and infrastructure automation.
- Provide technical mentorship and problem-solving support to the team.
- Proven experience as a Lead Engineer or similar role.
- Hands-on expertise in Azure services and PySpark.
- Strong experience in ETL development and database migrations.
- Excellent communication and leadership skills.
- Must have experience in the Insurance domain.