Senior Big Data Engineer (AWS | PySpark | EMR)
Summary
Senior Big Data Engineer at ITC Infotech in Bangalore (hybrid/onsite) building and optimizing enterprise-scale, TB/PB-volume data platforms on AWS. Day-to-day work centers on PySpark/Scala and Spark on EMR, ETL/ELT pipeline design, Airflow/Step Functions orchestration, and Spark job and cluster performance tuning.
Senior Big Data Engineer (AWS | PySpark | EMR)
Location: Bangalore (Hybrid/Onsite)
Experience: 10 to 16Years
Relevant Experience: Minimum 8 Years in Data Engineering / Big Data
Job Description
We are looking for a highly skilled Senior Big Data Engineer with strong hands-on experience in AWS, PySpark/Scala, Apache Spark, EMR, and enterprise-scale data processing.
Candidates should have proven experience building and optimizing large-scale distributed data platforms and must be comfortable working with complex ETL ecosystems handling massive data volumes.
Mandatory Skills
AWS (Must Have)
- Strong hands-on experience with AWS ecosystem
- Expert knowledge of:
- AWS EMR
- S3
- Glue
- Redshift
- Athena
- Lambda
- Step Functions
- CloudWatch
- Experience designing and managing large-scale AWS data platforms
Big Data Technologies (Must Have)
- Apache Spark
- PySpark and/or Scala
- Hadoop Ecosystem
- Hive
- HDFS
- Spark SQL
- Distributed Data Processing
Data Engineering (Must Have)
- Design and development of scalable ETL/ELT pipelines
- Batch and large-scale data processing
- Data Lake and Data Warehouse solutions
- Performance tuning of Spark applications
- Data quality and validation frameworks
Orchestration (Must Have)
- Apache Airflow
- AWS Step Functions
- Workflow orchestration
- DAG design and optimization
- Job scheduling and dependency management
- State management for enterprise data pipelines
Required Experience
- 8+ years of Data Engineering / Big Data experience
- Strong experience handling TB/PB-scale datasets
- Experience working with large Spark clusters
- Expertise in Spark optimization techniques:
- Partitioning
- Caching
- Broadcast joins
- Shuffle optimization
- Memory tuning
- Cluster tuning
- Experience solving distributed computing challenges
Preferred Skills
- Kafka
- Spark Streaming
- Databricks
- Snowflake
- CI/CD Pipelines
- Terraform
- Python
- SQL
- Cloud Migration Projects
Responsibilities
- Build and maintain scalable Big Data platforms on AWS
- Develop and optimize PySpark/Scala applications
- Design enterprise-grade ETL/ELT frameworks
- Create and maintain Airflow DAGs and orchestration workflows
- Optimize Spark jobs and EMR cluster performance
- Implement monitoring, alerting, and reliability solutions
- Work closely with business and analytics teams to deliver data solutions
- Troubleshoot production issues and improve platform stability