Senior AWS Data Engineer â Data Lake Platform
Description
Job Title : Senior AWS Data Engineer – Data Lake Platform
Experience : 5+ Years
Location: Gurugram Sector 43, Haryana
Work Mode : Work From Office (5 Days a Week – Monday to Friday)
Open Positions : 8+
About the Role :
We are looking for a skilled Senior AWS Data Engineer with strong hands-on experience in building, managing, and optimizing enterprise-scale data lake platforms on AWS.
The ideal candidate will have expertise in AWS data services, PySpark, Apache Airflow, and large-scale data pipeline development. The role involves designing end-to-end data ingestion and processing workflows, maintaining data quality, optimizing platform performance, and supporting analytics and downstream data consumption requirements.
Mandatory skills :
AWS Data Lake, Amazon S3, EMR, Glue, Athena, Redshift, DMS, Lambda, RDS, IAM, PySpark, Apache Spark, Apache Airflow, Python, Advanced SQL, Data Modeling, Production Data Pipelines, Hasura/GraphQL, PostgreSQL, and Production Support.
Key Responsibilities :
- Design, develop, and maintain scalable AWS-based data lake platforms and end-to-end data pipelines.
- Build data ingestion, staging, transformation, curated, and consumption layers using AWS services.
- Develop and optimize data processing pipelines using PySpark, Apache Spark, and AWS EMR.
- Implement workflow orchestration and scheduling using Apache Airflow.
- Work with Amazon S3, AWS Glue, Glue Data Catalog, Amazon Athena, and Amazon Redshift for data storage, cataloging, processing, and analytics.
- Develop efficient SQL queries and implement data modeling strategies for enterprise reporting and analytics.
- Build and support data-serving solutions using Amazon RDS, AWS Lambda, API Gateway, and PostgreSQL-based platforms.
- Monitor pipeline performance, troubleshoot production issues, perform root-cause analysis, and ensure data pipeline SLAs.
- Implement data security, access controls, PII protection, and governance best practices across the platform.
- Optimize data processing performance, query execution, storage utilization, and AWS infrastructure costs.
- Collaborate with application, analytics, and business teams to deliver reliable and scalable data solutions.
Mandatory Skills :
- AWS Data Services : Amazon S3, EMR, Glue, Athena, Redshift, DMS, Lambda, RDS, and IAM.
- Data Engineering : Strong hands-on experience building and maintaining enterprise data lake platforms.
- Programming : Python, PySpark, and Apache Spark.
- Orchestration : Production experience with Apache Airflow.
- Database & SQL : Advanced SQL, data modeling, and query optimization.
- Data Architecture : Experience with data ingestion, staging, transformation, curated layers, and consumption architecture.
- Production Support : Pipeline monitoring, troubleshooting, root-cause analysis, and SLA management.
- API / Data Serving : Experience with Hasura or a similar GraphQL-based data-serving layer integrated with PostgreSQL.
- Platform Ownership : Ability to independently manage data pipelines, improve platform reliability, and collaborate with cross-functional teams.
Good-to-Have Skills :
- Experience with dbt for data transformation and modeling.
- Knowledge of data lakehouse architectures and table formats such as Apache Iceberg, Delta Lake, or Apache Hudi.
- Experience with Amazon Redshift workload optimization, distribution keys, sort keys, and Spectrum.
- Knowledge of Kafka and real-time or streaming data ingestion.
- Experience with PostgreSQL or Amazon Aurora as a data-serving layer.
- Familiarity with data governance, compliance, and sensitive-data management.
- Experience in the BFSI or health insurance domain.
Ideal Candidate Profile :
- 5+ years of hands-on experience in AWS Data Engineering.
- Strong experience building and supporting production-grade AWS data lake platforms.
- Practical expertise in PySpark, Apache Airflow, SQL, and AWS data services.
- Ability to troubleshoot complex data pipeline issues and optimize performance.
- Strong understanding of data architecture, data modeling, and production support practices.
- Candidates with experience in enterprise data platforms and insurance or BFSI environments will be preferred.
Skills
- Airflow
- Analytics
- API
- API Gateway
- Athena
- Aurora
- AWS
- Aws Glue
- Data Engineering
- Data Governance
- Data Ingestion
- Data Lake
- Data Modeling
- Data Pipelines
- Data Quality
- dbt
- Delta Lake
- GraphQL
- IAM
- Iceberg
- Kafka
- Lakehouse
- Lambda
- PostgreSQL
- PySpark
- Python
- RDS
- Redshift
- Root Cause Analysis
- S3
- SLA Management
- Spark
- SQL
- Workflow Orchestration
Sign in to read and join the discussion.