AWS Data Engineer
Posted 4
views
Summary
A 6-month contract (view to extend) data engineering role in Abu Dhabi focused on building and optimizing scalable AWS-based data ingestion pipelines using Glue, PySpark, S3, Athena, and Lambda. The engineer implements CDC, full/delta and historical loading patterns, automates with Git and Terraform, and ensures pipelines meet data quality and contract standards.
Contract Length 6 Months view to extend Start Date ASAP
The role will focus on building scalable AWS-based ingestion pipelines optimizing performance and ensuring alignment with architecture and data quality standards
Key Responsibilities
- Build and maintain data ingestion pipelines using AWS Glue PySpark S3 Athena and Lambda Implement CDC full load delta snapshot and historical loading patterns
- Develop and optimize Bronze-layer ingestion pipelines
- Improve pipeline performance through partitioning parallelization and AWS best practices
- Automate deployment and infrastructure using Git and Terraform
- Ensure pipelines comply with data contracts monitoring and data quality standards
- Work closely with Data Architects Data Assurance and Platform teams
- Identify opportunities to automate repetitive migration and engineering activities
Required Skills & Experience
- 5+ years of experience in Data Engineering.
- Strong hands-on experience with AWS data services, particularly Glue, S3, Athena, and Lambda.
- Strong PySpark and ETL/ELT development experience.
- Good understanding of Lakehouse architecture and Medallion design principles.
- Experience with CDC, batch ingestion, delta loads, and historical data migration.
- Experience with Git, Terraform, CI/CD, and Agile/Jira.
- Banking or financial services experience is an advantage.