RedShift Data Engineer
Summary
Designs, builds, and tunes scalable ETL/ELT data pipelines and dimensional data models on Amazon Redshift and AWS (Glue, S3, Lambda, Step Functions), using Python and SQL. The role also covers performance tuning, data quality, CDC/incremental loading, and production support within an Agile team.
Job Overview
We are seeking an experienced Redshift Data Engineer with 8+ years of experience in data engineering and strong hands-on expertise with Amazon Redshift. The ideal candidate will be responsible for designing, developing, optimizing, and maintaining scalable data platforms and ETL/ELT pipelines on AWS.
Key Responsibilities
- Design and develop scalable data pipelines and ETL/ELT workflows using AWS services.
- Develop and optimize complex SQL queries, stored procedures, views, and data models in Amazon Redshift.
- Perform Redshift performance tuning, query optimization, distribution/sort key optimization, and workload management.
- Build and maintain data ingestion pipelines from multiple sources into Amazon Redshift.
- Work with AWS Glue, S3, Lambda, Step Functions, and CloudWatch for data integration and orchestration.
- Develop robust data transformation processes using Python and SQL.
- Implement data quality, validation, reconciliation, and monitoring frameworks.
- Design and maintain dimensional data models, including fact and dimension tables.
- Work with large datasets and optimize data processing for performance and cost.
- Implement incremental data loading and CDC (Change Data Capture) strategies.
- Troubleshoot data pipeline failures, performance issues, and production incidents.
- Collaborate with Data Architects, Business Analysts, Developers, and other stakeholders.
- Follow best practices for data security, governance, access control, and AWS IAM.
- Participate in code reviews, technical design discussions, and Agile delivery processes.
Mandatory Skills
- 8+ years of experience in Data Engineering
- Strong hands-on experience with Amazon Redshift
- Advanced SQL skills
- Strong Python programming/scripting experience
- Experience developing ETL/ELT pipelines
- Strong experience with AWS
- Hands-on experience with Amazon S3
- Experience with AWS Glue
- Experience with data warehousing and dimensional data modeling
- Strong knowledge of Redshift performance tuning
- Experience with large-scale data processing
- Experience with CI/CD and Git
- Good understanding of data quality and data governance
Good to Have
- Experience with AWS Lambda, Step Functions, EMR, Athena, and CloudWatch
- Experience with Apache Spark / PySpark
- Experience with Airflow
- Knowledge of Kafka / streaming data
- Experience with CDC tools such as AWS DMS, Debezium, or equivalent
- Experience with Terraform / Infrastructure as Code
- Exposure to Databricks, Snowflake, or other cloud data platforms
- Experience working in Agile/Scrum environments
Technical Skills
Cloud: AWS, S3, Glue, Lambda, Step Functions, CloudWatch
Database: Amazon Redshift, PostgreSQL
Programming: Python, SQL
ETL/ELT: AWS Glue, Airflow, custom Python pipelines
Big Data: Spark, PySpark
DevOps: Git, CI/CD, Terraform
Data Modeling: Star Schema, Snowflake Schema, Dimensional Modeling
Streaming: Kafka - desirable
Skills
- Agile
- Airflow
- Athena
- AWS
- Aws Glue
- CI/CD
- Cloud
- CloudWatch
- Data Engineering
- Data Governance
- Data Ingestion
- Data Modeling
- Data Pipelines
- Data Quality
- Data Warehousing
- Databricks
- Debezium
- DevOps
- Dimensional Modeling
- ELT
- ETL
- Git
- IAM
- Infrastructure as Code
- Kafka
- Lambda
- PostgreSQL
- PySpark
- Python
- Redshift
- S3
- Scrum
- Snowflake
- Spark
- SQL
- Terraform