Lead Data Engineer (AWS Data Platform)
Posted Updated
Description
NorthBay Solutions LLC seeks a Lead Data Engineer (AWS Data Platform) with 5–8+ years of experience to design, develop, and optimize large-scale data platforms and ETL/ELT pipelines. The role involves hands-on work with PySpark, AWS Glue, EMR, and Redshift, delivering robust data warehousing solutions.
The candidate will collaborate with a UAE-based customer and distributed teams, ensuring performance tuning, data quality, security, and governance. Hybrid work in Pakistan is offered.
Design, develop, and maintain scalable data pipelines using PySpark, AWS Glue, and Amazon EMR. Build and optimize data ingestion, transformation, and processing frameworks for structured and semi-structured data. Develop and maintain enterprise data warehouse solutions using Amazon Redshift. Write complex SQL queries, stored procedures, and data transformations to support analytics and reporting requirements. Implement ETL/ELT processes to move data efficiently across multiple systems and platforms. Perform performance tuning and optimization of Spark jobs, ETL pipelines, SQL queries, and Redshift workloads. Ensure data quality, integrity, security, and governance across data platforms. Troubleshoot production issues and perform root cause analysis for data-related incidents. 5 to 8+ years of experience in Data Engineering and Data Warehousing. Strong hands-on experience with PySpark. Extensive experience with AWS Glue. Experience building and managing workloads on Amazon EMR. Strong expertise in Amazon Redshift. Excellent SQL development and query optimization skills.Skills
Sign in to read and join the discussion.