Principal Databricks Data Engineer
Posted Updated
Description
Astra-North Infoteck Inc. is seeking an accomplished Data Engineering/Architecture lead in Toronto to drive the Cloudera to Databricks transformation. You will shape finance and risk data platforms and enable regulatory, management, and analytics reporting across the organization.
The role requires deep expertise in Databricks, Spark, data lakehouse architectures, and cloud platforms (AWS/Azure). You will mentor engineers, establish standards, and collaborate with governance stakeholders to
Lead Cloudera to Databricks transformation initiatives. Shape enterprise finance and risk data platforms. Support regulatory, management, and analytical reporting systems. 12–18 years of overall Data Engineering experience. 8+ years of enterprise Data Warehouse and Data Lake platform experience. 5+ years hands-on with Databricks and Apache Spark at scale. Modernize Cloudera platforms to Databricks Lakehouse architecture. Redesign ingestion, transformation, and consumption from HDFS-based architecture to Cloud Object Storage and Delta Lake. Refactor Hive and Impala logic into PySpark and Spark SQL ELT pipelines. Ensure data reconciliation, audit integrity, and consistency during migration. Design and govern enterprise Data Warehouse and Data Lake/Lakehouse architectures. Strong finance and risk data models including Financial Hierarchies and Liquidity Risk. Enable reporting: semantic/consumption layers, define metrics, dimensions, KPIs. Experience with AWS or Azure cloud platforms; CI/CD; data security and governance. dbt is an advantage.Skills
Sign in to read and join the discussion.