Senior Data Engineer

Job Title: Senior Data Engineer
Location: Noida
Employment Type: Full-time
Experience Required: 8 - 12 Years

About the Role/Project

The role involves maintaining and developing data pipelines, serverless migration, and CI/CD process ownership. The solution product line encompasses several key areas, including marketing analytics, category optimisation, pricing optimisation, customer experience improvement, and sales performance enhancement.

Key Responsibilities

• Adapt existing pipelines and code for serverless environments, ensuring stable execution
• Maintain and enhance CI/CD processes, including proposing architectural improvements
• Monitor pipelines and repositories, review merge requests, and oversee code changes
• Develop and maintain data pipelines while understanding underlying business logic

Required Skills & Qualifications

• 8 years of experience in Data Engineering
• Strong experience with Databricks and Delta Lake
• Strong experience designing Lakehouse Architecture
• Strong experience with PySpark and Spark SQL
• Experience building configuration-driven data pipelines.
• Strong Python programming skills with experience developing reusable Python libraries/packages.
• Experience with Delta table optimisation and maintenance.
• Recent project experience on the Databricks ecosystem is mandatory.
• Experience with serverless architectures
• Understanding of pricing or financial data domains

Preferred / Good to Have

• Experience with Azure Data Platform.
• Knowledge of CI/CD for data pipelines.
• Performance tuning and optimisation of large-scale data pipelines.

Technology Stack

• Databricks (Most important)
• Delta Lake
• PySpark
• Spark SQL
• Python
• Lakehouse Architecture
• Configuration-driven ETL/Data Pipeline Frameworks
• Python package/library development
• Delta Table optimisation & maintenance