Big Data Engineer
Posted Updated
Description
Roles and Responsibilities :
- Design, develop, test, deploy and maintain large-scale big data solutions using Hadoop, Spark, Hive and other related technologies.
- Collaborate with cross-functional teams to identify business requirements and design scalable architectures for data processing pipelines.
- Develop automated testing frameworks using Python to ensure high-quality delivery of big data projects on Azure/AWS platforms.
- Troubleshoot complex issues related to big data processing workflows and provide technical guidance to junior team members.
- Support day-to-day operations of Hadoop and Big Data platforms including HDFS, Nifi, YARN, Hive, Spark, Kafka, and ZooKeeper in non-prod and prod environments.
- Perform routine cluster administration tasks such as service restarts, monitoring, log analysis, and basic troubleshooting.
- Assist with Spark and Kafka pipeline support, issue triage, and incident resolution under guidance from senior engineers.
- Cloud platforms (AWS, Azure, PCF)
- Programming and scripting (Python, Go, Bash)
- Monitoring and observability tools
- DevOps and CI/CD practices
- Incident response and troubleshooting
- Execute standard operating procedures for upgrades, patching, certificate renewals, and access provisioning.
- Use Unix/Linux skills (shell scripting, process analysis, filesystem troubleshooting) to resolve platform issues.
- Contribute to automation efforts using Jenkins, Chef and/or Ansible, following existing playbooks and patterns.
- Work closely with application teams to onboard workloads and ensure adherence to platform standards.
- Support day-to-day operations of Hadoop and Big Data platforms including HDFS, Nifi, YARN, Hive, Spark, Kafka, and ZooKeeper in non-prod and prod environments.
Job Requirements :
- 4-5 years of experience in Big Data Engineering with expertise in Apache NiFi, HDFS, Kafka, Spark, Hive etc.
- Strong understanding of cloud-based technologies such as Microsoft Azure or AWS.
- Proficiency in programming languages like Python along with experience working with SRE tools like Jenkins.
Skills
Sign in to read and join the discussion.