Principal Data Engineer – Databricks
Summary
Leads the transformation of legacy Cloudera data platforms to Databricks Lakehouse architecture, designing scalable data pipelines and modernizing enterprise data warehouses with PySpark, Delta Lake, and cloud technologies.
Principal Data Engineer – Databricks | Spark | Delta Lake | PySpark | Data Lakehouse | AWS/Azure
Location: Toronto
Work Model: Onsite (4 days/week)
Key Requirements
- 12–18 years of overall Data Engineering experience.
- 8+ years of experience with Enterprise Data Warehouse and Data Lake platforms.
- 5+ years of hands‑on experience with Databricks and Apache Spark at scale.
- Strong experience modernizing legacy Cloudera platforms (CDH/CDP, Hive, HBase, Impala, Spark) to Databricks Lakehouse.
- Experience redesigning ingestion, transformation, and consumption patterns from HDFS-based architecture to cloud object storage and Delta Lake.
- Experience refactoring legacy Hive/Impala logic into PySpark and Spark SQL ELT pipelines.
- Experience ensuring data reconciliation, audit integrity, and consistency during migration.
- Experience designing and governing Enterprise Data Warehouse and Data Lake/Lakehouse architectures.
- Experience implementing layered architectures including:
- Raw/Landing Layer
- Curated/Conformed Layer
- Semantic/Consumption Layer
- Experience modernizing traditional Enterprise Data Warehouse platforms into scalable Lakehouse architectures.
- Strong experience with finance and risk data models, including:
- General Ledger
- Sub‑ledger
- Financial Hierarchies
- Credit Risk Models
- Liquidity Risk Models
- Market Risk Models
- Experience enabling reporting use cases including aggregation, drill‑down, and drill‑back capabilities.
- Experience building and managing semantic/consumption layers for BI, reporting, and analytics.
- Ability to define business metrics, dimensions, hierarchies, and KPIs.
- Experience with Databricks SQL, Delta Tables, and dbt or similar frameworks.
- Strong experience developing and optimizing large‑scale data pipelines using:
- PySpark
- Spark SQL
- Delta Lake
- Experience implementing Medallion Architecture:
- Bronze Layer
- Silver Layer
- Gold Layer
- Experience optimizing workloads using Z‑ORDER, OPTIMIZE, caching, and cluster configurations.
- Experience implementing data governance, data quality frameworks, reconciliation controls, and exception handling.
- Experience establishing data lineage and metadata management.
- Knowledge of data security, access control, and compliance standards.
- Experience with cloud platforms such as AWS or Azure.
- Experience with CI/CD pipelines using:
- Git
- Terraform
- Jenkins
- Azure DevOps
- Familiarity with orchestration tools such as:
- Apache Airflow
- Databricks Workflows
- Experience with dbt is a plus.
- Ability to act as a technical authority and lead architecture decisions.
- Experience mentoring senior engineers and establishing engineering standards.
- Strong stakeholder management skills with finance, risk, analytics, and governance teams.
- Ability to translate complex data structures into business‑ready insights.
Nice to Have
- Experience in Banking, Financial Services, Insurance (BFSI), Capital Markets, or regulatory reporting.
- Exposure to:
- SAP Finance
- Oracle Financials
- SAP S/4HANA
- Experience supporting AI/ML workloads.
- Databricks or cloud certifications.
Key Responsibilities
- Lead Cloudera to Databricks transformation initiatives.
- Design and implement enterprise Data Lakehouse and Data Warehouse solutions.
- Build scalable, high‑performance data pipelines and modern data architectures.
- Drive data modernization, governance, quality, and security initiatives.
- Support regulatory, management, and analytical reporting platforms.
- Provide technical leadership, mentor engineering teams, and establish best practices.