Senior Data Engineer
Summary
Senior Data Engineer (6+ yrs) building scalable data pipelines on Azure Databricks with Spark, Scala, Airflow, Hive, Iceberg, and Kubernetes, plus CI/CD via GitHub Actions and governance with Unity Catalog. Hybrid role (3 days/week in office) in Bengaluru-Whitefield, hired through Quess IT Staffing.
Job Description
Data Engineer – Azure Databricks
Experience: 6+ Years
Location: Bengaluru- Whitefield
Work Mode: Hybrid – 3 days/week
Interview Process: 2 Technical Round
Job Summary
We are looking for an experienced Data Engineer with 6+ years of hands-on experience in building and managing scalable data engineering solutions. The ideal candidate should have strong expertise in Spark, Scala, Airflow, Hive, Kubernetes, Iceberg, and Azure Databricks .
The candidate should also have a good understanding of modern data platforms, cloud technologies, CI/CD practices, and the ability to leverage AI across the Software Development Lifecycle (SDLC) and Data Engineering to drive innovation and business value.
Key Responsibilities
- Design, develop, and maintain scalable and reliable data engineering pipelines.
- Build data processing solutions using Apache Spark and Scala .
- Develop and manage workflows using Apache Airflow .
- Work with Hive, Iceberg, Kubernetes, and Docker in modern data platforms.
- Develop and maintain CI/CD pipelines using GitHub Actions .
- Design and implement data solutions using Azure Databricks .
- Work with ADLS, Azure Data Factory (ADF), Delta Lake, Databricks Jobs and Pipelines .
- Utilize Lakeflow Connect, Unity Catalog, and Databricks Compute (Classic & Serverless) .
- Implement data governance, security, and access controls using Unity Catalog.
- Optimize data pipelines and processing workloads for performance and scalability.
- Collaborate with data scientists, architects, product teams, and other engineering teams.
- Apply AI capabilities and tools across the SDLC and Data Engineering lifecycle to improve productivity, automation, and business outcomes.
- Follow engineering best practices around version control, testing, deployment, monitoring, and documentation.
Required Skills
Must-Have – Priority 1
- Strong hands-on experience with Apache Spark
- Scala
- Apache Airflow
- Hive
- Kubernetes
- Apache Iceberg
- Good knowledge of Docker
- Good experience with GitHub Actions / CI-CD
Must-Have – Priority 2
- Strong experience with Azure Databricks
- ADLS
- Databricks Jobs & Pipelines
- Azure Data Factory (ADF)
- Delta Lake
- Lakeflow Connect
- Unity Catalog
- Databricks Compute – Classic & Serverless
Must-Have – Priority 3
- Experience using AI tools/capabilities within SDLC and Data Engineering
- Ability to identify opportunities to use AI for automation, development, optimization, and innovation
- Strong problem-solving and analytical skills
Additional Requirements
- 6+ years of relevant experience in Data Engineering.
- Strong understanding of data engineering concepts and distributed data processing.
- Excellent communication and collaboration skills.
- Candidates must be based in or willing to relocate to Bengaluru .
- Candidates should be comfortable working from the office
- Willingness to attend 2 technical interview rounds .
Preferred Candidate Profile
Candidates with strong hands-on experience across Spark/Scala/Airflow and the Azure Databricks ecosystem will be given priority. Experience in modern data lakehouse technologies, containerization, CI/CD, and practical application of AI in engineering environments will be an added advantage.