Databricks Data Engineer
Posted Updated 2
views
Summary
adesso is hiring a Databricks Data Engineer in Pune to design, build, and maintain scalable batch and real-time data pipelines using Databricks, PySpark, Spark SQL, and Delta Lake (Medallion Architecture). The role involves workflow orchestration, Spark performance tuning, data quality/governance, and CI/CD on cloud platforms like Azure, AWS, or GCP.
YOUR ROLE
Design, develop, and maintain scalable batch and real-time data pipelines using Databricks, PySpark, and Spark SQL.
Build and implement Delta Lake-based solutions, including data ingestion, transformation, and Medallion Architecture (Bronze, Silver, Gold) layers.
Develop and orchestrate end-to-end data workflows using Databricks Workflows and other scheduling tools.
Optimize Spark workloads for performance, scalability, and cost efficiency through effective cluster management and tuning practices.
Ensure data quality, reliability, and governance through validation, monitoring, and testing frameworks.Integrate and process data from multiple enterprise sources, including databases, APIs, files, and streaming platforms.
Collaborate with cross-functional teams, including data scientists, analysts, architects, and business stakeholders, to deliver data-driven solutions.
Contribute to code management, CI/CD implementation, documentation, and continuous improvement initiatives.
YOUR PROFILE
4-6 years of experience in Data Engineering, with strong hands-on expertise in Databricks.
Proficiency in Python, PySpark, and SQL for building scalable data processing solutions.
Strong understanding of Apache Spark architecture, Delta Lake, and modern data engineering practices.
Experience working with cloud platforms such as Azure, AWS, or GCP.Solid knowledge of ETL/ELT processes, data modeling, and data warehousing concepts.
Experience with version control systems such as Git and CI/CD processes.
Familiarity with technologies such as Unity Catalog, Azure Data Factory, ADLS Gen2, Kafka, Event Hubs, Structured Streaming, or Snowflake is an advantage.
Exposure to Infrastructure-as-Code tools such as Terraform is a plus.Databricks certifications will be considered an added advantage.
Bachelor's degree in Computer Science, Information Technology, or a related field.