Big Data Engineer

Key Responsibilities

• Design, develop, and maintain big data pipelines leveraging Hadoop, Spark, and Kafka.

• Develop and optimize J2EE backend services for large-scale data processing and integration.

• Program in Python, Scala, and Java to build robust data and backend solutions.

• Architect and implement cloud-native solutions on GCP and Azure platforms.

• Design and maintain data models, data warehousing solutions, and BI tools to enable data-driven decision making.

• Implement and manage workflow orchestration frameworks such as Automic and Airflow for data pipeline automation.

• Collaborate with cross-functional teams including data scientists, analysts, and DevOps engineers to deliver end-to-end data solutions.

• Ensure data quality, security, and scalability across all platforms.

• Troubleshoot and optimize performance of large-scale distributed systems.

Required Skills & Qualifications

• Bachelor’s/Master’s degree in Computer Science, Engineering, or related field (or equivalent experience).

• 6–8years of experience in big data engineering and backend development.

• Expertise in Hadoop ecosystem (HDFS, MapReduce, Hive, Spark) and Kafka.

• Strong programming skills in Python, Scala, and Java.

• Hands-on experience with J2EE backend development.

• Proven experience with cloud services (GCP, Azure) for data engineering and application deployment.

• Proficiency in data modeling, warehousing concepts, and BI tools.

• Experience with workflow orchestration tools such as Airflow, Automic.

• Strong problem-solving and debugging skills in large-scale distributed environments.

Preferred Skills (Nice to Have)

• Familiarity with containerization and orchestration (Docker, Kubernetes).

• Knowledge of streaming frameworks (Spark Streaming, Flink, Storm).

• Exposure to CI/CD pipelines and DevOps practices.

• Experience in Agile/Scrum development environments.