Senior Data Engineer – Databricks
Summary
Build and optimize PySpark-based ETL pipelines on Databricks, refactoring legacy code and ensuring scalable, high-quality data flows for enterprise clients.
Essential Technical Skills
• Data Engineering: Strong foundation in data engineering principles, ETL/ELT processes, and data pipeline design patterns
• PySpark: Proven hands-on experience developing data pipelines using PySpark, including DataFrames API, Spark SQL, and performance optimization
• Databricks Platform: Practical experience with Databricks workspace, cluster management, notebooks, and job orchestration
• Workspace AI Agent: Knowledge of Databricks Workspace AI Agent capabilities and integration
• Data Modelling: Experience implementing data models including dimensional modeling, data vault, or lakehouse architectures
• Delta Lake: Understanding of Delta Lake features including ACID transactions, schema evolution, and optimization techniques
• Python: Strong Python programming skills for data processing and automation
Additional Technical Skills • SQL proficiency for data querying and transformation • Experience with cloud platforms (Azure, AWS, or GCP) • Understanding of data governance and security best practices • Knowledge of streaming data processing (Structured Streaming) • Familiarity with DevOps practices and CI/CD pipelines • Experience with version control systems (Git) • Understanding of data quality frameworks and testing methodologies
Professional Experience • Minimum 8 years in data engineering or related roles • At least 2-3 years of hands-on experience with Databricks platform • Proven track record of refactoring legacy code to modern frameworks • Experience building and maintaining production data pipelines at scale • Background working across multiple data sources and formats • Experience in agile development environments
Required Certifications - mandatory to have at least one certification •
Databricks Certified Data Engineer Associate OR Databricks Certified Data Engineer Professional
Additional Certifications (Preferred) • Databricks Certified Associate Developer for Apache Spark • Cloud platform certifications (Azure Data Engineer Associate, AWS Certified Data Analytics, or Google Cloud Professional Data Engineer) • Relevant data engineering or big data certifications
Soft Skills • Strong problem-solving and analytical thinking abilities • Excellent communication skills to explain technical concepts clearly • Ability to work collaboratively in cross-functional teams • Self-motivated with strong attention to detail • Adaptable to changing priorities and technologies • Client-focused mindset with commitment to quality delivery
Additional Technical Skills • SQL proficiency for data querying and transformation • Experience with cloud platforms (Azure, AWS, or GCP) • Understanding of data governance and security best practices • Knowledge of streaming data processing (Structured Streaming) • Familiarity with DevOps practices and CI/CD pipelines • Experience with version control systems (Git) • Understanding of data quality frameworks and testing methodologies
Professional Experience • Minimum 8 years in data engineering or related roles • At least 2-3 years of hands-on experience with Databricks platform • Proven track record of refactoring legacy code to modern frameworks • Experience building and maintaining production data pipelines at scale • Background working across multiple data sources and formats • Experience in agile development environments
Required Certifications - mandatory to have at least one certification •
Databricks Certified Data Engineer Associate OR Databricks Certified Data Engineer Professional
Additional Certifications (Preferred) • Databricks Certified Associate Developer for Apache Spark • Cloud platform certifications (Azure Data Engineer Associate, AWS Certified Data Analytics, or Google Cloud Professional Data Engineer) • Relevant data engineering or big data certifications
Soft Skills • Strong problem-solving and analytical thinking abilities • Excellent communication skills to explain technical concepts clearly • Ability to work collaboratively in cross-functional teams • Self-motivated with strong attention to detail • Adaptable to changing priorities and technologies • Client-focused mindset with commitment to quality delivery