AWS Data Engineer
Summary
Build and maintain scalable data pipelines using PySpark, AWS Glue, and Step Functions, plus serverless automation with AWS Lambda and IaC via Terraform.
Key Responsibilities
. Design, develop, and maintain scalable data pipelines using PySpark on AWS.
. Build and orchestrate ETL/ELT workflows using AWS Glue and AWS Step Functions .
. Develop serverless applications and automation using AWSLambda .
. Write clean, efficient, and maintainable Python/PySpark codefollowing engineering best practices.
. Provision and manage cloud infrastructure using Terraform (Infrastructure as Code).
. Implement and maintain CI/CD pipelines to automate codedeployment, testing, and infrastructure changes.
. Monitor, troubleshoot, and optimize data pipelines forperformance, reliability, and cost efficiency.
. Collaborate with business stakeholders to deliver datasolutions.
. Follow DevOps, security, and coding standards throughout theengagement.
Required Skills. Strong hands-on experience with PySpark and Python fordata engineering.
. Experience developing ETL pipelines using AWS Glue .
. Proficiency with AWS Step Functions for workfloworchestration.
. Experience building serverless solutions using AWS Lambda .
. Hands-on experience with Terraform for Infrastructure asCode (IaC).
. Experience implementing CI/CD pipelines using tools suchas GitLab, GitHub Actions, Jenkins, or similar.
. Good understanding of AWS services, data lakes, IAM, S3,CloudWatch, and monitoring.