Azure Data Bricks Consultant
JOB SUMMARY
Design, develop, and optimize scalable data engineering solutions using Azure Databricks. Build end-to-end ETL/ELT pipelines and Lakehouse architectures. Implement data governance, security, and performance optimization best practices. Collaborate with cross-functional teams in a consulting environment to deliver enterprise data solutions.
Key Responsibilities
- Design and develop ETL/ELT pipelines using Azure Databricks.
- Build batch ingestion using Auto Loader and real-time ingestion using Spark Structured Streaming.
- Develop and maintain Delta Lake-based data pipelines.
- Manage Unity Catalog for data governance, security, and access control.
- Create and maintain catalogs, schemas, tables, materialized views, functions, and volumes.
- Develop Slowly Changing Dimensions (SCD Type 1 & Type 2) for dimensional data.
- Build Change Data Capture (CDC) pipelines for incremental data processing.
- Implement Lakehouse Federation and configure foreign catalogs for external data integration.
- Optimize data storage using partitioning and Liquid Clustering.
- Ensure data quality, integrity, and consistency across data platforms.
- Participate in CI/CD implementation and follow DevOps best practices.
- Collaborate with business and technical stakeholders to deliver scalable data solutions.
Required Qualifications
- 7–10 years of experience in Data Engineering.
- Hands‑on experience with Azure Databricks and Delta Lake.
- Strong expertise in ETL/ELT pipeline development.
- Experience with Spark Structured Streaming and Auto Loader.
- Strong knowledge of Unity Catalog.
- Proficiency in SQL and data modeling.
- Strong understanding of Data Warehousing concepts.
- Experience with Microsoft Azure cloud services.
- Excellent analytical and problem‑solving skills.