Senior Data Engineer
Summary
Design and deliver data pipelines and ETL/ELT processes for client-facing analytics, architecting and optimising data models and transformations in Databricks using Python, Spark, and SQL at scale.
Data Engineering Architect - Data Engineer
- Lead the design and delivery of data pipelines and ETL/ELT processes that support the Client-Facing Analytics (ACIA) reporting and analytical capabilities.
- Architect, develop and optimise data models, transformations, and pipeline solutions scaling to multi-terabyte datasets in Databricks, using Python, Spark, and SQL.
- Tune Spark performance across memory management, caching strategies, and cluster resource optimisation; manage Databricks cluster configuration for performance and cost efficiency.
- Partner with the Lead and stakeholders to translate business requirements into scalable, secure, production-ready solutions aligned to corporate design and layered architecture standards.
- Drive engineering standards and best practice across Git/GitHub workflows, CI/CD pipelines, testing, and modern web application deployment.
- Own end-to-end module delivery with minimal supervision, identifying and mitigating technical risks early and escalating dependencies and blockers through the agreed governance model.
- Ensure data quality, lineage, and observability across pipelines, addressing issues identified through monitoring and reviews.
- Apply Git/GitHub workflows and CI/CD practices to data engineering deliverables; mentor junior engineers and maintain up-to-date technical and operational documentation
- Champion the adoption of AI-assisted development practices (including vibe-coding) within the team to accelerate delivery without compromising quality.
Location of Work:
Bangalore India (on-site + remote)
On-site expectations: 2-3 days per week
Working Hours: 12:00-20:00 IST