Senior Databricks Data Engineer

About the Role:

We are seeking a Senior Databricks Data Engineer to design and develop scalable data pipelines, optimize Spark workloads, and build enterprise-grade data engineering solutions using Azure Databricks. The engineer will play a critical role in migrating Finance data from on-premises SAP systems into a modern cloud Lakehouse architecture.

Key Responsibilities

• Design and develop scalable ETL/ELT pipelines.
• Develop PySpark and Spark SQL transformations.
• Build Bronze, Silver, and Gold layer data pipelines.
• Integrate SAP and enterprise source systems.
• Implement Delta Lake best practices.
• Optimize Spark jobs for performance and cost.
• Build reusable data engineering frameworks.
• Implement data quality and validation rules.
• Develop incremental and CDC pipelines.
• Collaborate with BI developers for optimized reporting datasets.
• Participate in code reviews and technical documentation.
• Support production deployments and troubleshooting.

Roles & Responsibilities

Ideal Candidate

• 812 years of Data Engineering experience.
• Strong hands-on experience with Azure Databricks.
• Expert in PySpark and Spark SQL.
• Experience with Delta Lake.
• Experience with Azure Data Factory.
• Knowledge of ADLS Gen2.
• Strong SQL skills.
• Experience with Git and CI/CD.
• Understanding of SAP data extraction techniques.

Preferred Certification

• Databricks Certified Data Engineer Professional.
• Experience with Structured Streaming.
• Knowledge of MLflow.
• Azure DevOps experience.
• Experience with Power BI datasets.