Data Engineer
Posted Updated
Summary
Data engineer in Mumbai building and maintaining batch and real-time data pipelines, ETL workflows, and data warehouses/lakes. Core stack is Python, SQL, PySpark, and AWS data services (Glue, S3, Lambda, Redshift, Athena), with BFSI/NBFC/FinTech experience preferred.
Key Responsibilities
- Develop and maintain batch and real-time data pipelines to support business requirements.
- Build and maintain ETL workflows for data ingestion, transformation, and integration.
- Work with data warehouses and data lakes to support data storage and processing requirements.
- Monitor pipeline execution, investigate data discrepancies, and resolve technical issues.
- Perform unit testing, data validation, and code debugging to ensure data accuracy and reliability.
- Collaborate with senior engineers, analysts, and business teams to deliver data engineering solutions.
- Follow established coding standards, documentation practices, and deployment procedures.
Mandatory Requirements
- 3+ years of relevant experience in Data Engineering.
- Strong hands-on proficiency in Python programming and OOP concepts.
- Good knowledge of SQL, including complex queries, joins, and query optimization.
- Practical experience with Apache Spark / PySpark for distributed data processing.
- Hands-on experience with AWS data services such as AWS Glue, Amazon S3, Lambda, Redshift, and Athena.
- Understanding of ETL/ELT processes, data warehouses, and data lakes.
- Familiarity with data quality, data validation, and data governance principles.
- Working knowledge of Git and Agile development practices.
Preferred
- Exposure to AWS EMR, Glue Data Catalog, and Lake Formation.
- Experience in BFSI, NBFC, or FinTech.
- Familiarity with JIRA, Confluence, Jenkins, or dashboard tools such as Tableau and Amazon QuickSight.