Databricks Engineer
Posted Updated
Description
About the Role
We are looking for a skilled Databricks Engineer with hands-on experience in designing, developing, and optimizing scalable data engineering solutions using the Databricks Lakehouse Platform.
The ideal candidate should have strong expertise in Python, PySpark, SQL, Apache Spark, and Delta Lake, with experience building end-to-end data pipelines and processing large volumes of structured and unstructured data.
Key Responsibilities
- Design, develop, and maintain scalable data engineering solutions using the Databricks platform.
- Build and optimize ETL/ELT pipelines using PySpark, Python, and Spark SQL.
- Develop data ingestion, transformation, and processing workflows using Delta Lake.
- Implement and manage Delta Tables, including MERGE operations, ACID transactions, and time travel.
- Work on Databricks notebooks, jobs, workflows, and Lakehouse architecture.
- Optimize Spark applications for performance, scalability, and cost efficiency.
- Implement data quality checks, validation, error handling, and monitoring.
- Integrate Databricks with cloud storage and services across Azure, AWS, or GCP.
- Troubleshoot production issues and ensure data pipeline reliability.
- Collaborate with Data Architects, Data Engineers, and business stakeholders.
- Follow best practices for data governance, security, version control, and CI/CD.
Mandatory Technical Skills
- Strong hands-on experience with the Databricks platform.
- Proficiency in Python and PySpark programming.
- Strong knowledge of Apache Spark architecture and distributed data processing.
- Expertise in SQL, including complex queries, joins, window functions, and query optimization.
- Hands-on experience with Delta Lake and Delta Tables.
- Experience in developing and maintaining ETL/ELT data pipelines.
- Knowledge of data modelling, partitioning, and performance optimization.
- Experience working with at least one cloud platform: Azure, AWS, or GCP.
- Experience with Databricks Workflows or equivalent orchestration tools.
Certifications Preferred
- Databricks Certified Data Engineer Associate.
- Databricks Certified Data Engineer Professional.
Professional certification will be an added advantage.
Skills
Sign in to read and join the discussion.