Software Engineer II (Data Engineering)

Job: Software Engineer II (Data Engineering)
Location: Bangalore

Responsibilities:
- Design, develop, and maintain scalable data processing pipelines using the Spark ecosystem, including
PySpark, Databricks, and Synapse Notebooks.
- Develop and manage data pipelines using Azure Data Factory, Azure Synapse, and Azure Data Lake to
ingest, process, and integrate data from Azure and on-premises sources.
- Implement full and incremental data load strategies, ensuring efficient, reliable, and timely movement of
data across source systems and target platforms.
- Perform data cleansing, transformation, aggregation, and modeling to prepare high-quality datasets for
analytics, reporting, and downstream consumption.
- Apply Azure security, access control, and data governance practices to ensure secure and compliant data
processing and storage.
- Optimize Spark jobs, data pipelines, and SQL workloads for performance, scalability, reliability, and cost
efficiency.
- Collaborate with BI and analytics teams to support Power BI reports and dashboards, including
datasets/models hosted on Azure Analysis Services (AAS) and Power BI.
- Troubleshoot data pipeline and processing issues, perform root-cause analysis, and implement appropriate
fixes to maintain data availability and quality.
- Develop reusable frameworks, processes, and best practices for data ingestion, transformation, validation,
and monitoring.
- Work closely with business stakeholders, data analysts, and engineering teams to translate data
requirements into scalable and maintainable data solutions.

Key Skills Required:
Data engineering & Analytics:
- 5+ years of experience in statistics and analytics on large-scale datasets
- Strong SQL and data analysis skills, including SQL DW concepts
- Experience with Spark ecosystem: PySpark, Databricks, Synapse Notebooks (1+ year hands-on)
- Proficiency in Azure Data Services:
o Azure Synapse
o Azure Data Factory
o Azure Data Lake
o Azure security model
- Experience building pipelines for full and incremental data loads from Azure and on-prem sources
- Strong knowledge of data management, processing, cleaning, transformation, aggregation, and modeling
- Working knowledge of Power BI reports and dashboards (AAS / PBI)
Systems, Cloud & DevOps
- Strong programming skills in Python (Java is a plus)
- Experience with FastAPI, RESTful services, and microservices architecture
- Proficiency with databases such as MySQL, PostgreSQL, and MongoDB
- Experience deploying applications using Docker and Kubernetes
- Experience with streaming and messaging systems: Kafka, Kinesis, RabbitMQ
- Familiarity with Azure Git and DevOps practices
Leadership & Soft Skills
- Proven experience leading projects end-to-end
- Strong mentoring and team leadership experience
- Excellent written and verbal communication skills
- Ability to work effectively in distributed teams
- Strong understanding of data structures and design patterns
- Exposure to the retail industry is a plus

Good to have skills:
- AI /ML and Agentic AI

Education:
- Bachelor's degree in computer science or a related field (required)
- Master's degree is a plus