DataBricks - Data Engineering

Job Description

Role Purpose

The purpose of this role is to facilitate application solutions by developing and reviewing module-level codes, creating replication patterns, performing root cause analysis to identify recurring issues and collaborating with internal and external stakeholders to ensure stable, well-documented deliverables that meet business objectives.

J͏ob Role:-

About the Role

Highly skilled Senior Databricks Platform Engineer to help build and scale a next-generation data platform supporting analytics, AI, governance, and self-service data consumption. The role focuses on engineering reusable platform capabilities, enabling enterprise-scale governance, improving observability, and driving operational excellence across the Databricks ecosystem.

Key Responsibilities

· Design and develop scalable data pipelines using Databricks, PySpark, SQL, and Delta Lake.

• Build CDC, batch, and streaming ingestion frameworks for enterprise-scale workloads.
• Implement and manage Medallion Architecture (Bronze, Silver, Gold).
• Develop governance solutions using Unity Catalog, including row-level security, data masking, and fine-grained access controls.
• Build and maintain Data Quality, Monitoring, and Observability frameworks.
• Implement CI/CD pipelines using Databricks Asset Bundles (DABS), Git, and Infrastructure as Code.
• Optimize Databricks clusters, SQL Warehouses, and workloads for performance and cost efficiency.
• Enable self-service analytics through semantic layers, metric views, governed datasets, and certified data products.
• Partner with platform, security, analytics, and business teams to drive adoption of Databricks capabilities.
• Participate in architecture discussions, platform modernization initiatives, and stakeholder workshops.

S͏kills:-

Required Skills

Databricks & Lakehouse

· Databricks Lakehouse Platform

• Delta Lake
• Unity Catalog
• Databricks Workflows
• Auto Loader
• Structured Streaming
• Databricks SQL

Data Engineering

· PySpark

• Advanced SQL
• Data Modeling
• ETL/ELT Frameworks
• CDC Architectures
• Real-Time Data Processing

Nice to have: -

Preferred Skills (These are not a must, nice to have)

· Scala

• dbt
• Kafka
• Airflow
• GenAI / AI Agents
• Enterprise Observability Solutions