Big Data Engineer
Posted
Description
Job Title: Senior Big Data Engineer / Hadoop Spark Lead
Experience: 10-15 Years
Location: Bangalore / Chennai
Role: Senior Technical / Lead Role
Department: IT Data Engineering
Industry: IT Services & Consulting
Job Summary
We are looking for an experienced Senior Big Data Engineer with strong expertise in the Hadoop Ecosystem, PySpark, SQL, Linux and Shell Scripting. The candidate will be responsible for designing, developing and optimizing scalable big data solutions and data processing pipelines.
Key Responsibilities
- Design, develop and maintain large-scale data processing solutions using the Hadoop Ecosystem.
- Develop and optimize data pipelines using PySpark and SQL.
- Work on Linux environments and automate processes using Shell Scripting.
- Handle data processing, transformation, integration and performance optimization.
- Troubleshoot complex data engineering issues and ensure data quality.
- Collaborate with cross-functional teams to deliver scalable data solutions.
Mandatory Skills
- Strong experience in Core Big Data / Hadoop Ecosystem.
- Hands-on experience in PySpark.
- Strong SQL skills.
- Linux / Unix.
- Shell Scripting.
Good-to-Have Skills
- Kubernetes
- DevOps
- Apache Airflow
- MinIO
- Apache Iceberg