Senior Data Engineer - Databricks

Senior Data Engineer

Location: Noida, India

Work format: Office

Experience: 8+ years in Data Engineering, including 2+ years with Databricks

Project:

Our customer is one of the TOP-5 global management consulting firms. The role is a backfill for a departing senior data engineer in the client's Solution Product Line (SPL). It involves maintaining and developing data pipelines, serverless migration and ownership of the CI/CD process.

Requirements:

- Bachelor's or Master's degree in Computer Science, Engineering or a related field

- 8+ years of experience in Data Engineering with strong big data expertise

- Deep expertise in Databricks (2+ years), Spark and SQL

- Strong data modeling and ETL experience with large-scale datasets

- Hands-on experience orchestrating data pipelines with Airflow and CI/CD (GitHub Actions or similar)

- Proven experience with Delta Live Tables, Unity Catalog, Delta Sharing and SQL Warehouse

- Proficiency in Python for data engineering workflows

- Familiarity with analytics engineering and dbt

- Experience with data governance, compliance and security in a cloud environment

- Experience adopting software engineering best practices (testing, modularization, refactoring, automation)

- Ability to optimize data systems for performance and cost efficiency in cloud environments

- Demonstrated ability to lead projects independently and mentor junior engineers

- English: Upper-Intermediate or higher

Responsibilities:

- Serverless migration: adapt existing pipelines / code to run on different cluster types (serverless) without issues

- CI/CD ownership: maintain and improve CI/CD processes, propose architectural improvements

- Pipeline monitoring: proactively check repositories, review merge requests and team members' changes

- Functional and technical work: write pipeline code and understand the business logic (pricing domain)