Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

InfoBeans

NewBe an early applicant

Senior Data Engineer - Azure Databricks

Posted Updated

Description

Job Description:

Senior Data Engineer Azure Databricks

Job Overview:

We are looking for an experienced Senior Data Engineer with strong hands-on expertise in Azure Databricks, PySpark, Apache Spark, Delta Lake and Azure data services. The role will focus on designing and developing scalable data engineering solutions, building batch and streaming pipelines, implementing Lakehouse architectures, and optimizing Spark workloads for performance, reliability and maintainability.

Key Responsibilities:

- Design, develop and maintain scalable data pipelines using Azure Databricks, PySpark and Spark SQL.

- Build and manage Lakehouse solutions using Bronze, Silver and Gold / Medallion architecture.

- Develop robust batch and streaming ETL/ELT pipelines for structured and semi-structured data.

- Work extensively with Delta Lake, including MERGE/upserts, schema evolution, data quality and incremental processing.

- Implement and manage data governance using Unity Catalog, including catalogs, schemas, permissions and access controls.

- Develop and orchestrate Databricks notebooks, jobs and workflows for reliable data processing.

- Integrate Databricks with Azure services such as ADLS Gen2, Azure Data Factory, Synapse Analytics, Event Hubs and Service Bus.

- Optimize Spark workloads using appropriate partitioning, caching, broadcast joins, shuffle optimization and other performance-tuning techniques.

- Apply Spark fundamentals such as lazy evaluation, transformations/actions, joins, repartitioning and data-skew handling.

- Design data models and curated datasets for analytics, reporting and downstream applications.

- Implement data quality checks, schema validation, monitoring, error handling and SLA-driven pipeline delivery.

- Develop solutions using Python and SQL and follow clean, modular and maintainable coding practices.

- Use Git, Azure DevOps and CI/CD practices for source control, deployment and release management.

- Collaborate with engineering, analytics and business teams to understand requirements and deliver reliable data products.

Required Technical Skills:

- Programming: Python, SQL

- Big Data: Apache Spark, PySpark, Spark SQL

- Databricks: Azure Databricks, notebooks, jobs/workflows, performance optimization

- Lakehouse: Delta Lake, Medallion architecture, incremental processing, MERGE, schema evolution

- Governance: Unity Catalog, access control, data governance

- Azure Data: ADLS Gen2, Azure Data Factory, Azure Synapse Analytics

- Messaging / Integration: Azure Event Hubs, Azure Service Bus

- DevOps: Git, Azure DevOps, CI/CD, Docker

- Databases: SQL Server, PostgreSQL, Oracle, Azure SQL

Qualifications & Experience:

- 8+ years of overall experience in software/data engineering.

- Strong hands-on experience in Azure Databricks and PySpark-based data engineering.

- Strong Python and SQL development skills.

- Demonstrated experience designing and supporting production-grade data pipelines.

- Strong analytical, troubleshooting and problem-solving skills.

- Good communication and ability to explain technical approaches clearly.

Skills

What Senior Data Engineering jobs ask for — and how much of it you have →
Apply

See also

Data Engineering jobs by country — openings, pay and top skills →