Pyspark Data Engineer

Required Experience

• 3-5 years of hands-on experience in data engineering, with a strong track record of delivering production grade data pipelines.
• Experience in the banking or financial services domain is highly preferred.
• Experience implementing data solutions on cloud-native platforms is an advantage.

Technical Skills

• Strong hands-on proficiency in PySpark for large-scale data processing on Cloudera Data Platform (CDP).
• Experience with distributed data processing frameworks on CDP.
• Solid understanding in PySpark Coding, debugging and Productionization skills.
• Exposure to AI Assisted Coding tools like Claude Code is a huge advantage.
• Solid understanding of Data Vault 2.0 and dimensional modelling techniques.
• Familiarity with data warehousing concepts, Big Data architectures/ components and technologies (Hive, Spark, Oozie, Airflow, etc.) is a must.
• Experience working with CI/CD pipelines and version control (e.g., Git, Jenkins).
• Knowledge of data quality frameworks, metadata management, and data governance practices.
• Proficiency in Exploratory Data Analysis (EDA) to understand data characteristics and inform pipeline design.
• Exposure to ETL and BI reporting tools such as Microsoft Power BI is an advantage.
• Experience with Feature engineering for analytical or ML use cases is an added advantage.
• Familiarity with GitHub Actions for pipeline automation is a plus