Senior Databricks Data Engineer
Posted Updated 6
views
Summary
Senior Databricks Data Engineer at adesso in Bangalore: designs enterprise-scale Databricks Lakehouse solutions, builds batch and real-time pipelines with PySpark, Spark SQL, and Structured Streaming, governs data via Unity Catalog, and drives CI/CD, DevOps, and platform optimization while mentoring engineers.
YOUR ROLE
- Design and deliver enterprise-scale data solutions using Databricks Lakehouse architecture and Delta Lake.
- Lead the development of scalable batch and real-time data pipelines using PySpark, Spark SQL, and Structured Streaming.
- Implement and govern data security, lineage, access management, and compliance standards using Unity Catalog.
- Drive platform performance optimization, scalability improvements, and cost management initiatives.
- Establish and enhance CI/CD pipelines, DevOps practices, automated testing frameworks, and Infrastructure-as-Code implementations.
- Provide technical leadership through code reviews, mentoring, and engineering best practices.
- Partner with architects, business stakeholders, and delivery teams to translate business requirements into scalable data solutions.
- Troubleshoot and resolve complex production issues while ensuring platform reliability and operational excellence.
- Support solution architecture, technical estimations, framework development, and project delivery activities.
YOUR PROFILE
- 6-8 years of experience in Data Engineering, including significant hands-on experience with Databricks.Advanced expertise in Python, PySpark, Spark SQL, and large-scale data processing frameworks.
- Strong experience with Delta Lake, Unity Catalog, Databricks Workflows, and Delta Live Tables.
- Proven experience designing and implementing cloud-based data platforms on Azure, AWS, or GCP.In-depth understanding of data modeling, data warehousing, and modern Lakehouse architectures.
- Experience working with real-time data technologies such as Kafka, Event Hubs, and Structured Streaming.Strong knowledge of Git, DevOps practices, CI/CD pipelines, and Infrastructure-as-Code methodologies.
- Demonstrated ability to lead technical initiatives, mentor team members, and collaborate with global stakeholders.
- Exposure to MLflow, Microsoft Fabric, Azure Synapse, Snowflake, and enterprise data governance frameworks is desirable.
- Bachelor's or Master's degree in Computer Science, Information Technology, or a related discipline.