Hadoop Lead - Data Engineering

Job Summary :

Seeking a Hadoop Developer with experience in big data engineering to design, build, and optimize scalable data pipelines on Hadoop ecosystem technologies. The role involves batch/stream processing, data integration, performance tuning, and close collaboration with business teams.

Key Responsibilities :

- Design and develop robust ETL/data pipelines using Hadoop ecosystem tools.

- Build and optimize large-scale data processing jobs using Spark and/or Hive.

- Develop workflows using Airflow.

- Write complex SQL/HiveQL for data transformation and reporting needs.

- Ingest structured and unstructured data from multiple sources (Kafka, Sqoop, APIs, files).

- Ensure data quality, lineage, and governance standards are followed.

- Monitor, troubleshoot, and tune jobs for performance and scalability.

- Create technical documentation and follow coding best practices.

Required Skills :

- Hands-on experience in Hadoop/big data development.

- Strong knowledge of HDFS, Hive, Spark.

- Strong knowledge in Python/Scala.

- Strong SQL and data modelling fundamentals.

- Experience with workflow orchestration tools (Airflow).

- Good understanding of Linux, shell scripting, and distributed systems.

- Familiarity with version control (Git) and Agile delivery practices.