Azure Data Bricks Consultant
Posted Updated
Description
Compunnel, Inc. in Toronto is seeking an experienced Data Engineer to design, develop, and optimize scalable data solutions using Azure Databricks.
You will build end-to-end ETL/ELT pipelines and Lakehouse architectures, collaborating with cross‑functional teams in a consulting environment to deliver enterprise data solutions. The role requires 7–10 years of experience with Azure Databricks, Delta Lake, and SQL/data modeling, focusing on governance, security, and performance optimization in a
Design and develop ETL/ELT pipelines using Azure Databricks. Build batch ingestion using Auto Loader and real-time ingestion using Spark Structured Streaming. Develop and maintain Delta Lake-based data pipelines. Manage Unity Catalog for data governance, security, and access control. Create and maintain catalogs, schemas, tables, materialized views, functions, and volumes. Develop Slowly Changing Dimensions (SCD Type 1 & Type 2) for dimensional data. Build Change Data Capture (CDC) pipelines for incremental data processing. Implement Lakehouse Federation and configure foreign catalogs for external data integration. Optimize data storage using partitioning and Liquid Clustering. Ensure data quality, integrity, and consistency across data platforms. Participate in CI/CD implementation and follow DevOps best practices. Collaborate with business and technical stakeholders to deliver scalable data solutions. 7–10 years of data engineering experience. Hands-on experience with Azure Databricks and Delta Lake. Strong expertise in ETL/ELT pipeline development. Experience with Spark Structured Streaming and Auto Loader. Strong knowledge of Unity Catalog. Proficiency in SQL and data modeling. Strong understanding of Data Warehousing concepts. Experience with Microsoft Azure cloud services. Excellent analytical and problem‑solving skills.Skills
Sign in to read and join the discussion.