Databricks Data Engineer

About the RoleClient is seeking an experienced Databricks Data Engineer to design, develop, and maintain scalable data pipelines and cloud-based data platforms. The ideal candidate will have expertise in Databricks, PySpark, Delta Lake, and modern data engineering practices to support enterprise analytics, reporting, and business intelligence initiatives.Key Responsibilities

• Design, build, and optimize scalable ETL/ELT pipelines using Databricks and PySpark.
• Develop and maintain data ingestion frameworks for batch and real-time data processing.
• Implement and manage Delta Lake, Delta Live Tables, and Lakehouse architecture solutions.
• Work with Azure/AWS cloud services for data storage, processing, and orchestration.
• Transform and integrate structured and unstructured data from multiple enterprise sources.
• Develop reusable data models and frameworks to support enterprise analytics.
• Optimize Spark jobs, cluster configurations, and query performance.
• Implement data quality checks, monitoring, and governance controls.
• Collaborate with business stakeholders, architects, and analysts to deliver data solutions.
• Ensure data security, compliance, and best practices across the data ecosystem.
• Support CI/CD processes and automation for data platform deployments.

Required Skills

• 5+ years of Data Engineering experience.
• Strong hands-on experience with Databricks.
• Expertise in PySpark, Spark SQL, Python, and SQL.
• Experience with Delta Lake, Delta Live Tables (DLT), Unity Catalog.
• Strong understanding of Data Lake, Lakehouse, Medallion Architecture (Bronze, Silver, Gold).
• Experience with Azure Databricks or AWS Databricks.
• Experience with Azure Data Factory (ADF), ADLS Gen2, or equivalent AWS services.
• Strong knowledge of data modeling, ETL/ELT concepts, and performance tuning.
• Experience with Git, CI/CD pipelines, and Agile methodologies.
• Excellent problem-solving and communication skills.

Preferred Skills

• Experience with Snowflake, Synapse Analytics, Redshift, or Fabric.
• Knowledge of Kafka, Event Hubs, or other streaming technologies.
• Exposure to Unity Catalog and data governance frameworks.
• Experience with Power BI, Tableau, or other BI tools.
• Databricks certifications are highly preferred.
• Manufacturing, Banking, Retail, or Enterprise Applications domain experience.

Nice to Have

• Databricks Certified Data Engineer Associate/Professional.
• Knowledge of Data Governance, Data Lineage, and Data Quality frameworks.
• Experience in cloud migration and modernization projects.
• Exposure to AI/ML data pipelines and MLOps.

Education

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or related field.

Keywords for SourcingDatabricks, PySpark, Spark SQL, Delta Lake, Delta Live Tables, DLT, Unity Catalog, Azure Databricks, AWS Databricks, ADF, ADLS, Synapse, Snowflake, Data Lake, Lakehouse, ETL, ELT, Data Engineering, Python, SQL.