AWS Data Engineer
Summary
Build and maintain scalable data pipelines and ETL workflows on AWS using PySpark, Glue, Lambda, and Step Functions, while automating infrastructure with Terraform and CI/CD.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using PySpark on AWS.
- Build and orchestrate ETL/ELT workflows using AWS Glue and AWS Step Functions.
- Develop serverless applications and automation using AWS Lambda.
- Write clean, efficient, and maintainable Python/PySpark code following engineering best practices.
- Provision and manage cloud infrastructure using Terraform (Infrastructure as Code).
- Implement and maintain CI/CD pipelines to automate code deployment, testing, and infrastructure changes.
- Monitor, troubleshoot, and optimize data pipelines for performance, reliability, and cost efficiency.
- Collaborate with business stakeholders to deliver data solutions.
- Follow DevOps, security, and coding standards throughout theengagement.
Required Skills
- Strong hands-on experience with PySpark and Python fordata engineering.
- Experience developing ETL pipelines using AWS Glue.
- Proficiency with AWS Step Functions for workfloworchestration.
- Experience building serverless solutions using AWS Lambda.
- Hands-on experience with Terraform for Infrastructure as Code (IaC).
- Experience implementing CI/CD pipelines using tools such as GitLab, GitHub Actions, Jenkins, or similar.
- Good understanding of AWS services, data lakes, IAM, S3, CloudWatch, and monitoring.