Streaming Data Engineer- Bangalore
Summary
Senior engineer in Bengaluru who designs, operates, and migrates real-time data platforms on Confluent Platform/Cloud and Apache Kafka — owning topics, Connect/Debezium pipelines, Flink or Kafka Streams processing, schema governance, security, and cluster monitoring, plus migrations via MirrorMaker 2 and Cluster Linking.
Senior Streaming Data Engineer (Confluent / Apache Kafka)
Experience: 3-8 years
Location: [Bangalore]
Employment type: Full-time
About the Role
We are looking for a Senior Streaming Data Engineer to design, operate, and migrate real-time data platforms on Confluent Platform and Confluent Cloud. You will own critical pipelines that move data between applications, databases, and analytics systems, and lead migrations into and out of Confluent.
Experience with other streaming platforms (Kinesis, Pub/Sub, Event Hubs, Pulsar) is valued, as we want someone who can cross-skill quickly and map new platforms onto Kafka.
Responsibilities
- Design and manage Kafka solutions on Confluent: topic design, partitioning strategy, retention, replication, quotas, and ACLs/RBAC
- Lead end-to-end migrations into and out of Confluent (from Kinesis, Pub/Sub, or self-managed Kafka) using MirrorMaker 2, Replicator, Cluster Linking, or custom bridge consumers, including validation and rollback planning
- Guarantee ordering, idempotency, and consumer offset continuity during cutovers
- Build and review producers, consumers, and Kafka Connect pipelines (Debezium CDC included)
- Own schema governance with Schema Registry, including compatibility policies and evolution
- Develop stream processing applications using Flink, Kafka Streams, or ksqlDB
- Design CI/CD and infrastructure-as-code pipelines for consumers, connector configs, and cluster provisioning
- Monitor cluster health and consumer lag, and tune throughput, latency, and cost
- Implement security: encryption, authentication, and private networking
- Define architecture standards, write runbooks, and mentor junior engineers
Requirements
- 5+ years in data, platform, or distributed systems engineering, including 3+ years with Kafka / Confluent in production
- Deep understanding of Kafka internals and semantics: partitions, consumer groups, rebalancing, offsets, ordering, and exactly-once delivery
- Proven experience with Kafka administration, capacity planning, and troubleshooting production incidents
- Proficiency in Java or Python, strong SQL, and shell scripting
- Strong working knowledge of AWS or GCP (IAM, networking, storage), including VPC, TLS, and private connectivity
- Strong Linux fundamentals: debugging, networking commands, disk and memory behavior
- CI/CD experience (Git with Jenkins, GitHub Actions, or GitLab CI) and monitoring tools (Prometheus, Grafana, Datadog, or CloudWatch)
- Understanding of distributed systems fundamentals: replication, leader election, backpressure, and failure handling
Bonus Skills
Hands-on Kinesis or Pub/Sub, Go, Terraform (Confluent provider), Kubernetes (Confluent for Kubernetes, Strimzi), Snowflake/BigQuery, Confluent certification (Developer or Administrator)
