Data Engineer -RedShift
Summary
Senior data engineer (8+ years) designing, building, and tuning ETL/ELT pipelines and dimensional data models on Amazon Redshift and AWS (Glue, S3, Lambda, Step Functions), using Python and SQL. The role is with XPT Software, an IT services firm that places engineers at client sites across Australia.
- XPT SoftwareAustralia PTY Ltd, incorporated in 2016, is a Software Services company
- XPT works with topclients across Australia in Banking, Insurance, Telecom,Retail, Energy, Mining and Manufacturingdomains.
- We have 120+technocrats in Australia working at our clientlocations.
- XPT SoftwareAustralia is part of group companies which has globalpresence across India & Europe.
- We have served100+ clients globally, fulfilling their onsite-offshoreneeds.
We are seeking an experienced Redshift Data Engineer with 8+ years of experience in data engineering and strong hands‑on expertise with Amazon Redshift. The ideal candidate will be responsible for designing, developing, optimizing, and maintaining scalable data platforms and ETL/ELT pipelines on AWS.
Key Responsibilities- Design and develop scalable data pipelines and ETL/ELT workflows using AWS services.
- Develop and optimize complex SQL queries, stored procedures, views, and data models in Amazon Redshift.
- Perform Redshift performance tuning, query optimization, distribution/sort key optimization, and workload management.
- Build and maintain data ingestion pipelines from multiple sources into Amazon Redshift.
- Work with AWS Glue, S3, Lambda, Step Functions, and CloudWatch for data integration and orchestration.
- Develop robust data transformation processes using Python and SQL.
- Implement data quality, validation, reconciliation, and monitoring frameworks.
- Design and maintain dimensional data models, including fact and dimension tables.
- Work with large datasets and optimize data processing for performance and cost.
- Implement incremental data loading and CDC (Change Data Capture) strategies.
- Troubleshoot data pipeline failures, performance issues, and production incidents.
- Collaborate with Data Architects, Business Analysts, Developers, and other stakeholders.
- Follow best practices for data security, governance, access control, and AWS IAM.
- Participate in code reviews, technical design discussions, and Agile delivery processes.
- 8+ years of experience in Data Engineering
- Strong hands‑on experience with Amazon Redshift
- Advanced SQL skills
- Strong Python programming/scripting experience
- Experience developing ETL/ELT pipelines
- Strong experience with AWS
- Hands‑on experience with Amazon S3
- Experience with AWS Glue
- Experience with data warehousing and dimensional data modeling
- Strong knowledge of Redshift performance tuning
- Experience with large‑scale data processing
- Experience with CI/CD and Git
- Good understanding of data quality and data governance
- Experience with AWS Lambda, Step Functions, EMR, Athena, and CloudWatch
- Experience with Apache Spark / PySpark
- Experience with Airflow
- Knowledge of Kafka / streaming data
- Experience with CDC tools such as AWS DMS, Debezium, or equivalent
- Experience with Terraform / Infrastructure as Code
- Exposure to Databricks, Snowflake, or other cloud data platforms
- Experience working in Agile/Scrum environments
Cloud: AWS, S3, Glue, Lambda, Step Functions, CloudWatch
Database: Amazon Redshift, PostgreSQL
Programming: Python, SQL
ETL/ELT: AWS Glue, Airflow, custom Python pipelines
Big Data: Spark, PySpark
DevOps: Git, CI/CD, Terraform
Data Modeling: Star Schema, Snowflake Schema, Dimensional Modeling
Streaming: Kafka – desirable
Skills
- Agile
- Airflow
- Athena
- AWS
- Aws Glue
- CI/CD
- Cloud
- CloudWatch
- Data Engineering
- Data Governance
- Data Ingestion
- Data Modeling
- Data Pipelines
- Data Quality
- Data Warehousing
- Databricks
- Debezium
- DevOps
- Dimensional Modeling
- ELT
- ETL
- Git
- IAM
- Infrastructure as Code
- Kafka
- Lambda
- PostgreSQL
- PySpark
- Python
- Redshift
- S3
- Scrum
- Snowflake
- Spark
- SQL
- Terraform