Data Engineer (Azure Databricks)
Summary
Build and maintain Azure Databricks pipelines and Delta Lake tables, integrate with ADLS Gen2, and optimize ETL/ELT workflows for scalable data solutions.
Responsibilities:
Azure Databricks Platform Implementation
Develop and maintain data solutions using Azure Databricks (notebooks, Delta Lake)
Configure and optimize Databricks jobs, clusters, and workloads
Integrate with Azure Data Lake (ADLS Gen2)
Databricks Genie Implementation Support
Support implementation of Databricks Genie based on predefined architecture
Prepare curated datasets for Genie consumption
Optimize data structures for query performance
Data Pipeline Development
Build ETL/ELT pipelines using Databricks and Azure tools
Develop ingestion pipelines from APIs, databases, and external systems
Ensure pipelines are reliable, monitored, and production-ready
Data Management & Governance Apply data quality checks and controls Implement access control and data organization practices Document pipelines and datasets
Production Readiness & Optimization Ensure scalability and performance Optimize queries and pipeline efficiency. Improve Genie performance to deliver faster, more efficient results Skills & Experiences: Hands-on skills of Azure Databricks (Delta Lake, notebooks), Azure Data Lake (ADLS Gen2), Azure Data Factory or Synapse Solid SQL and ETL experience and medallion architecture familiarity Exposure to Databricks Genie, familiarity with Python/PySpark, Unity Catalog or data governance tools are advantageous Solid execution focus with the ability to deliver high-quality work independently High attention to details, particularly in data quality and accuracy Solid problem-solving skills, with the ability to troubleshoot data and pipeline issues Receptive to feedback and able to iterate quickly based on technical guidance Able to collaborate effectively within a technical team environment
Data Management & Governance Apply data quality checks and controls Implement access control and data organization practices Document pipelines and datasets
Production Readiness & Optimization Ensure scalability and performance Optimize queries and pipeline efficiency. Improve Genie performance to deliver faster, more efficient results Skills & Experiences: Hands-on skills of Azure Databricks (Delta Lake, notebooks), Azure Data Lake (ADLS Gen2), Azure Data Factory or Synapse Solid SQL and ETL experience and medallion architecture familiarity Exposure to Databricks Genie, familiarity with Python/PySpark, Unity Catalog or data governance tools are advantageous Solid execution focus with the ability to deliver high-quality work independently High attention to details, particularly in data quality and accuracy Solid problem-solving skills, with the ability to troubleshoot data and pipeline issues Receptive to feedback and able to iterate quickly based on technical guidance Able to collaborate effectively within a technical team environment