Big Data Engineer

Job Summary :

We are looking for an experienced Big Data Engineer to design, develop, and maintain scalable data pipelines capable of processing large volumes of structured and unstructured data. The ideal candidate should have strong expertise in distributed computing frameworks and modern big data technologies.

Key Responsibilities :

- Design and develop scalable Big Data solutions.

- Build and optimize ETL/ELT pipelines.

- Process large datasets using distributed computing frameworks.

- Develop reusable data ingestion and transformation frameworks.

- Optimize data storage, processing, and query performance.

- Collaborate with Data Scientists, Analysts, and Product teams.

- Ensure data quality, governance, and security best practices.

- Troubleshoot production issues and improve pipeline reliability.

Required Skills:

- 6-11 years of Data Engineering experience.

- Strong hands-on experience with Hadoop ecosystem.

- Expertise in Apache Spark, Hive, HDFS.

- Experience with Kafka or other streaming technologies.

- Strong SQL skills.

- Experience with Python or Scala.

- Knowledge of Airflow/Oozie or similar orchestration tools.

- Experience working on cloud platforms is an advantage.

Preferred Skills:

- CI/CD pipelines

- Docker & Kubernetes

- Data Warehousing concepts

- Agile methodology