Lead Data & BI Engineer

Experience: 8–12 years (with strong Databricks depth)

Employment Type: Contract - Full time

We are looking for a lead, hands-on Databricks Lead / Lakehouse Architect who can drive project kickoff, define the implementation strategy, and lead end-to-end delivery of an Azure Databricks Lakehouse along with Power BI semantic modeling and reporting for hospitality analytics. This role will own architecture, technical decisions, delivery planning, stakeholder alignment, and initial build-out, and then help ramp the broader team for implementation and rollout.

Key Responsibilities

• Lead discovery workshops, clarify business objectives, define scope, milestones, and delivery approach.
• Lead discovery workshops with hospitality business/tech stakeholders (Revenue Management, Operations, Finance, Loyalty, Digital).
• Define target-state lakehouse architecture and delivery roadmap (phased onboarding of domains: reservations, stays, POS, loyalty, digital).
• Create end-to-end implementation strategy: architecture, design standards, data onboarding plan, governance, and operating model.
• Define best practices, coding standards, branching strategy, CI/CD approach, and environments (dev/test/prod).
• Design and implement Azure Databricks Lakehouse architecture using Delta Lake and Medallion (Bronze/Silver/Gold).
• Design data ingestion patterns for batch (and streaming if needed) from varied sources (APIs, DBs, files, event streams).
• Build scalable transformation frameworks using PySpark / Spark SQL with performance and cost optimization.
• Implement and review complex pipelines: incremental loads, CDC patterns, SCD Type 1/2, data quality rules, and reconciliation.
• Own performance tuning: partitioning, file sizing, Z-ORDER, caching, adaptive query execution, cluster sizing, autoscaling.
• Implement monitoring/alerting, job orchestration, dependency management, and restart/recovery strategies.

Security, Governance & Operations

• Define security model: RBAC/ACLs, service principals, Key Vault integration, secrets management.
• Implement Unity Catalog strategy (if applicable): catalogs, schemas, lineage, permissions, data sharing.
• Establish operational readiness: runbooks, SLAs, logging, audit, cost governance.
• Act as primary technical point of contact for client stakeholders.
• Produce design documentation (HLD/LLD), implementation roadmap, and knowledge transfer plan.
• Mentor engineers, conduct code reviews, and ensure delivery quality.

Must-Have Skills

• Deep hands-on Databricks experience in end-to-end implementations (not only development).
• Strong experience with Delta Lake (MERGE, OPTIMIZE, VACUUM, schema evolution, time travel).
• Experience designing lakehouse architecture (medallion layers, curated zones, semantic datasets).
• Power BI development experience is mandatory.
• Strong understanding of:
• Data quality frameworks, reconciliation, and observability

Nice-to-Have (Strong Plus)

• Unity Catalog implementation experience (access control, lineage, governance).
• CI/CD for Databricks: Azure DevOps/GitHub, Databricks Repos, asset bundles, Terraform.
• Exposure to BI semantic layers (Power BI)
• Databricks certifications (Data Engineer Associate/Professional).
• AI/Copilot Integration experience is plus