Jr. Data Engineer

Responsibilities:

• Design, develop, and maintain ETL pipelines for data ingestion, transformation, and processing.
• Work with PySpark and Python to process large datasets efficiently.
• Optimise Redshift queries and manage data warehouse performance.
• Ensure data quality and integrity in the pipeline.
• Collaborate with data scientists, analysts, and software engineers to provide clean and reliable data.
• Automate workflows and monitor job execution for failures.
• Implement best practices in data engineering, performance tuning, and cost optimisation.

Requirements:

• 2+ years of experience in ETL pipeline development using Python and SQL.
• Hands-on experience with PySpark for big data processing.
• Strong knowledge of AWS Redshift, including query optimisation and performance tuning.
• Proficiency in SQL for complex queries, transformations, and aggregations.
• Good understanding of data modelling and best practices in data architecture.
• Strong problem-solving and debugging skills.
• Experience with workflow orchestration tools like Airflow is a plus.