Data Engineer – Scala / Spark / Kafka
Summary
A 6–12 month contract Data Engineer role in Bangalore designing and building scalable batch and real-time data pipelines. Day-to-day work centers on Scala, Apache Spark (including Structured Streaming), and Kafka, alongside Hive, SQL, S3, Presto, and Delta Lake on cloud data infrastructure.
Role: Data Engineer – Scala / Spark / Kafka
Location: India
Employment Type: Contract
Experience: 5+ Years
Duration: 6–12 Months
Start Date: ASAP
Job Description
We are looking for an experienced Data Engineer to design, build, and optimize scalable data platforms capable of processing large volumes of batch and real-time data.
The ideal candidate will have strong hands-on experience with Scala, Apache Spark, Kafka, and Big Data technologies. You will be responsible for developing robust data pipelines, designing scalable data models, troubleshooting performance issues, and collaborating with engineering and analytics teams to ensure reliable access to high-quality data.
The role involves extensive work with distributed processing technologies, streaming platforms, and modern cloud-based data infrastructure.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines for batch and real-time processing.
- Develop data processing applications using Scala and Apache Spark.
- Build and maintain real-time streaming pipelines using Kafka and Spark Structured Streaming.
- Design and implement scalable data models and data processing solutions.
- Work with Hive, SQL, S3, Presto, Delta Lake, and other Big Data technologies.
- Analyze and optimize Spark jobs and distributed data workloads.
- Troubleshoot data pipeline, performance, and production issues.
- Ensure data quality, reliability, scalability, and availability.
- Collaborate with engineering, analytics, and other technical teams to deliver data solutions.
- Contribute to the design and development of modern data lake and lakehouse architectures.
Required Skills
- 5+ years of professional experience in Data Engineering, Big Data Engineering, or a related field.
- Strong programming skills in Scala.
- Hands-on experience with Apache Spark for both batch and streaming workloads.
- Strong proficiency in Hive and SQL.
- Hands-on experience with Kafka and real-time event streaming.
- Strong understanding of Big Data technologies and ecosystems.
- Experience with S3, Hive, Presto, and Delta Lake.
- Experience designing and implementing scalable data models.
- Strong debugging and problem-solving skills.
- Experience with performance analysis and optimization of distributed data workloads.
- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.