Databricks Developer

Job Description:

We are seeking a skilled Senior Databricks Developer with strong expertise in Databricks, PySpark, Apache Spark, and cloud-based data engineering. The ideal candidate should have experience designing and implementing enterprise-grade Lakehouse solutions and delivering end-to-end data engineering projects.

Key Responsibilities:

• Design and develop scalable Lakehouse architectures using Azure Databricks, Delta Lake, and Unity Catalog.
• Build and optimize ETL/ELT pipelines using PySpark, SQL, Python, Auto Loader, and Delta Live Tables (DLT).
• Develop batch and streaming data processing solutions.
• Optimize Spark workloads through partitioning, caching, AQE, Catalyst Optimizer, and performance tuning.
• Implement Databricks Workflows, Serverless computing, and Databricks SQL solutions.
• Manage data governance, security, access controls, and metadata using Unity Catalog.
• Implement CI/CD pipelines using Git, Azure DevOps, GitHub, Jenkins, or GitLab.
• Work with Terraform and Databricks Asset Bundles for infrastructure and deployment automation.
• Support MLOps practices using MLflow, Model Registry, and model deployment.
• Collaborate with customers and stakeholders on requirements, architecture design, technical presentations, and solution delivery.

Mandatory Skills:

• 5+ years of overall IT experience with 4+ years in Data Engineering / Big Data.
• Strong hands-on experience with Databricks, PySpark, Apache Spark, and Delta Lake.
• Experience with Unity Catalog, Auto Loader, Databricks Workflows, DLT/Lakeflow, Databricks SQL, and Serverless.
• Strong understanding of Spark internals, including DAG, Driver/Executors, partitioning, shuffle, AQE, and Catalyst Optimizer.
• Experience with ETL/ELT, Python, SQL, data modeling, and batch/streaming pipelines.
• Strong expertise in AWS, Azure, or GCP, with working knowledge of another cloud platform.
• Experience in Spark performance tuning, Photon, cluster sizing, autoscaling, and cost optimization.
• Knowledge of Git, CI/CD, Terraform, and Databricks Asset Bundles.
• Experience with MLflow and MLOps practices.
• Databricks Certified Data Engineer Professional certification preferred.

Additional Requirements:

• Must have delivered 34 end-to-end Databricks implementations.
• Strong customer-facing communication and stakeholder management skills.
• Experience in enterprise data architecture, migration, and modernization is an advantage.