Senior Data Engineer
We are looking for a Senior Data Engineer to join a fast-paced, data-driven environment on a contract basis. This role is ideal for an engineer with deep hands-on expertise across the Azure and Databricks ecosystem who can design, build, and optimize enterprise-scale data platforms. You'll play a key role in modernizing ETL/ELT processes, strengthening data governance, and driving cost and performance efficiency across the Databricks Lakehouse.
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines using Azure and Databricks to support enterprise data and analytics needs
- Build and manage Delta Live Tables (DLT) pipelines for reliable, automated data transformation workflows
- Implement and maintain Unity Catalog for centralized data governance, access control, and lineage tracking across the Lakehouse
- Design and develop semantic layers to support consistent, business-friendly access to curated data for analytics and BI consumption
- Build and maintain CI/CD pipelines for data engineering workflows, ensuring automated testing, deployment, and version control best practices
- Own FinOps practices within Databricks — monitoring, forecasting, and optimizing compute/storage costs across clusters and jobs
- Conduct performance tuning and optimization of Databricks workloads, including cluster configuration, query optimization, and job orchestration
- Work across core Azure data services (e.g., Azure Data Factory, Azure Data Lake Storage, Azure Synapse) to integrate and orchestrate data pipelines
- Partner with data architects, analysts, and business stakeholders to translate requirements into robust, scalable data solutions
- Ensure data quality, security, and compliance standards are upheld across all pipelines and platforms
- Troubleshoot and resolve pipeline failures, performance bottlenecks, and data quality issues in a timely manner
Required Skills & Experience
- Proven experience as a Senior Data Engineer, with a strong track record delivering production-grade data pipelines
- Strong hands-on experience with Microsoft Azure (Azure Data Factory, ADLS, Azure Synapse, or similar)
- Deep expertise in Databricks, including:
- Unity Catalog
- Cluster and job performance optimization
- FinOps / cost management within Databricks
- Strong experience building and maintaining CI/CD pipelines for data engineering workflows (e.g., Azure DevOps, GitHub Actions)
- Experience designing and implementing semantic layers for analytics/BI consumption
- Solid background in ETL/ELT pipeline design, development, and optimization
- Proficiency in Python and/or SQL for data transformation and pipeline development
- Strong understanding of data governance, security, and lineage best practices
- Excellent problem-solving skills and ability to work independently in a contract setting
- Strong communication skills, with the ability to work cross-functionally with technical and business stakeholders
Nice to Have
- Databricks certifications (e.g., Databricks Certified Data Engineer Professional)
- Experience with Terraform or other infrastructure-as-code tools in an Azure/Databricks context
- Experience in a regulated or enterprise-scale industry
- Familiarity with Spark internals and advanced performance tuning