Senior Data Engineer (Databricks, Lakehouse & AI-BI)
Posted
Summary
Senior data engineer owning Databricks Lakehouse pipelines end to end — Delta Lake tables, Unity Catalog governance, Lakeflow orchestration and CDC — with Azure Data Factory for legacy sources, plus AI/BI Genie semantic layers for self-service analytics. Onsite in Abu Dhabi, ~6 years' experience.
Title: Senior Data Engineer (Databricks Lakehouse & AI-BI)
Location: ABU DHABI
Exp: 6 Years
Job Description:
Key Responsibilities
- 1. Pipeline Orchestration: Design and build robust data ingestion and orchestration pipelines with Databricks (Lakeflow Jobs/Workflows) as the primary platform and Azure Data Factory supporting legacy/external needs taking full ownership of pipeline reliability and the transition toward Databricks-native orchestration.
- 2. Lakehouse Data Engineering: Build and maintain scalable Delta Lake tables under Unity Catalog schema design access controls lineage tracking and data quality across bronze/silver/gold layers.
- 3. Lakeflow Pipeline Development: Build declarative ETL pipelines using Databricks Lakeflow incorporating CDC incremental processing and data quality validations.
- 4. AI/BI & Genie Enablement: Design and maintain semantic layers metrics and certified datasets to support Genie spaces and AI/BI dashboards enabling natural-language querying and self-service analytics.
- 5. SharePoint/External Source Integration: Configure and maintain Lakeflow Connect (or similar) integrations for ingesting SharePoint and other external data sources into the Lakehouse with minimal manual effort.
- 6. Governance & Best Practices: Apply Unity Catalog governance controls access policies data masking audit logging and uphold strong engineering practices (CI/CD version control testing).
Must-Have Skills
- Strong hands-on Databricks expertise PySpark/SQL Workflows/Jobs orchestration Delta Lake Unity Catalog
- Working knowledge of Azure Data Factory
- Experience with Lakeflow (Declarative Pipelines/Connect) including CDC and data quality validations
- Exposure to Databricks AI/BI or Genie (semantic layers certified datasets)
- Familiarity with CI/CD and version control (Git Databricks Repos/Asset Bundles)
- Demonstrated ability to own deliverables fully and resolve issues independently
- Strong communication skills for working with business/analytics stakeholders
Required Experience:
Senior IC