Lead Data Engineer
Description
Role: Lead Data Engineer
Experience: 8+ Years
Role SummaryWe are seeking a hands-on Technical Leader with 8+ years of experience to spearhead the modernization and future-state architecture of our Sales Validation platform. In this role, you will own technical delivery across major workstreams, drive our strategic migration from Azure Synapse to Databricks, and establish scalable engineering standards for our data ecosystem.
Key ResponsibilitiesArchitecture & Migration: Lead the end-to-end architecture and delivery of the Databricks Lakehouse platform, driving the migration of legacy workloads from Azure Synapse to Databricks.
Engineering Standards: Establish best-in-class engineering standards, governance patterns, and reusable code frameworks across data pipelines.
Technical Leadership: Own technical delivery across major workstreams, ensuring high reliability, scalability, and maintainability.
Mentorship & Quality: Mentor junior and mid-level data engineers, conduct thorough architecture/design reviews, and enforce code quality through peer reviews.
Optimization: Continuously monitor, troubleshoot, and drive performance tuning and cost scalability across Spark jobs and data storage layers.
Databricks Ecosystem (Highest Priority)
Deep hands-on expertise in Databricks and Lakehouse Architecture.
Strong proficiency in PySpark, Spark SQL, and Delta Lake.
Experience implementing Unity Catalog for data governance and access control.
Proficiency in orchestration and deployment using Databricks Workflows and Databricks Asset Bundles (DABs).
Migration Experience
Proven track record in the modernization of legacy ETL workloads.
Direct hands-on experience migrating from Azure Synapse components, including Synapse Pipelines, Synapse Spark, Synapse SQL Pools, Synapse Notebooks, and Synapse Data Flows.
Azure Cloud Platform
Strong working knowledge of Azure Data Factory (ADF), Azure Data Lake Storage (ADLS Gen2), and Azure SQL.
Solid understanding of Azure Security Patterns (RBAC, Managed Identities, Private Endpoints, Key Vault integration).
Data Engineering & DevOps
Advanced SQL skills and strong Data Modeling principles (dimensional modeling, Data Vault, or medallion architecture).
Hands-on experience with GitLab CI/CD pipelines for automated testing and deployment.
Proven expertise in Spark and SQL Performance Tuning and rigorous Code Reviews.
Skills
Sign in to read and join the discussion.