Data Engineer Pyspark and Azure
Posted
Summary
Lead Data Engineer role in Ontario, Canada focused on improving an insurance investment reporting system: building ETL pipelines and cloud-native architecture with PySpark and Azure (Data Factory, Databricks, Functions, Event Grid, Service Bus), plus DevOps/CI-CD practices and team mentorship.
We are seeking a highly skilled Data Engineer with expertise in PySpark and Azure to lead the improvement of our Investment Reporting System.
This role involves guiding the engineering team, implementing cloud-native solutions, and ensuring adherence to best practices.
Key Responsibilities
- Lead the development of ETL pipelines and cloud-native architectures using Azure services.
- Work hands-on with Azure Data Factory, Databricks, Functions, Event Grid, and Service Bus.
- Collaborate with Solution Architects to translate designs into actionable tasks.
- Implement DevOps practices, CI/CD pipelines, and infrastructure automation.
- Provide technical mentorship and problem-solving support to the team.
Required Qualifications
- Proven experience as a Lead Engineer or similar role.
- Hands‑on expertise in Azure services and PySpark.
- Strong experience in ETL development and database migrations.
- Excellent communication and leadership skills.
- Must have experience in the Insurance domain.