Data Engineer
Posted Updated
Description
Venquis is seeking an AWS Glue Data Engineer to join the Data Factory Squad in Abu Dhabi/Dubai. You will build scalable ingestion pipelines, using AWS Glue and PySpark, to migrate source systems into the Lakehouse ingestion zone.
You will implement CDC, full/delta loads, and historical loading strategies, optimize performance, and automate deployments via Git and Terraform. Collaboration with Data Architect and Data Assurance is essential.
Data Ingestion Development: Build and implement AWS Glue jobs for Bronze layer ingestion using defined standards and templates. Design and execute historical loading mechanisms to bring legacy data into the Lakehouse. Performance Optimization: Optimize Glue job performance (DPU allocation, parallelization, partitioning). Migration & Automation: Migrate source tables to Bronze layer with templates, later using AI-enabled acceleration; ensure version control and automated deployment via Git and Terraform. Governance & Monitoring: Ensure jobs comply with data contracts and are properly monitored; prepare documentation. Strong experience in AWS Glue, PySpark, and ETL pipeline development. Knowledge of Lakehouse architecture and Medallion design principles. Familiarity with CDC, delta loads, and historical data ingestion strategies.Skills
Sign in to read and join the discussion.