Databricks Data Engineer, Risk Technology
Summary
Build and maintain scalable data pipelines on Azure Databricks using PySpark, ingesting structured and unstructured data to power risk analytics and reporting for a large bank.
You will design, develop, and maintain scalable batch and streaming data pipelines using Azure Databricks and Apache Spark. You will build and optimize ETL/ELT frameworks to ingest structured and unstructured data into cloud platforms, and engineer curated datasets for downstream applications and business intelligence tools. You will document the software delivery lifecycle and follow secure coding practices.
Responsibilities
- Design, develop and maintain scalable batch and streaming data pipelines
- Build and optimize ETL/ELT frameworks
- Ingest data from structured and unstructured source systems into cloud platforms
- Engineer and publish curated datasets
- Document each step of the software delivery lifecycle
- Write secure code following industry best practices
- Express technical concepts to non-technical audiences
- Build trusted working relationships with stakeholders and vendors
- Apply banking and risk management knowledge
- Stay up-to-date with emerging technologies and regulatory developments
Requirements
- 5-7 years of Software Engineering experience
- Relevant post-secondary degree
- Proficiency in Python
- Hands-on experience with Databricks platform implementing ETL processes
- Comfort writing SQL queries
- Experience with PySpark or Spark SQL
- Experience with Microsoft Azure
- Experience with databases, cloud computing, or application development/support
- C# and .NET experience is an asset
- Exposure to risk management, capital stress testing, credit risk models, and regulatory compliance
Benefits
- Competitive salary
- Incentive pay
- Banking benefits
- Benefits program
- Vacation offering
- Wellbeing support
- MomentMakers social recognition program
- Medical insurance
- Dental insurance
- Vision insurance
- Health Savings Account
- Life insurance
- Disability insurance
- Paid Time Off
- Parental Leave
- Holidays
- 401(k)