Confluent Integration Engineer
Job Description
- Design, develop, and maintain Kafka-based data integration pipelines using Confluent Platform / Confluent Cloud
- Migrate existing custom connectors to Confluent Managed Connectors (S3 Sink/Source, JDBC, Debezium, REST, etc.)
- Configure, deploy, and monitor Kafka Connect clusters and connector instances
- Implement Schema Registry governance (Avro, JSON Schema, Protobuf) for data contracts
- Build and optimise Kafka Streams or ksqlDB applications for real-time data transformation
- Collaborate with the EIDH architecture team on integration patterns (event-driven, CDC, batch offload)
- Troubleshoot connector failures, consumer lag, partition rebalancing, and throughput issues
- Implement observability: monitoring, alerting, and dashboarding for Kafka ecosystem components
- Support data governance, lineage, and security (ACLs, RBAC, encryption in transit/at rest)
- Participate in incident response and production support for streaming infrastructure
Key Responsibilities
- Design, develop, and maintain Kafka-based data integration pipelines using Confluent Platform / Confluent Cloud
- Migrate existing custom connectors to Confluent Managed Connectors (S3 Sink/Source, JDBC, Debezium, REST, etc.)
- Configure, deploy, and monitor Kafka Connect clusters and connector instances
- Implement Schema Registry governance (Avro, JSON Schema, Protobuf) for data contracts
- Build and optimise Kafka Streams or ksqlDB applications for real-time data transformation
- Collaborate with the EIDH architecture team on integration patterns (event-driven, CDC, batch offload)
- Troubleshoot connector failures, consumer lag, partition rebalancing, and throughput issues
- Implement observability: monitoring, alerting, and dashboarding for Kafka ecosystem components
- Support data governance, lineage, and security (ACLs, RBAC, encryption in transit/at rest)
- Participate in incident response and production support for streaming infrastructure
Required Skills & Experience
Must Have
3+ years hands-on experience with Apache Kafka / Confluent Platform / Confluent Cloud
Strong experience with Kafka Connect — deploying, configuring, and managing connectors (both self-managed and Confluent Managed)
Experience migrating from custom/self-managed connectors to Confluent Managed Connectors
Proficiency with Schema Registry and schema evolution strategies
Experience with at least 3 of the following connectors: S3 Sink/Source, JDBC Source/Sink, Debezium CDC, HTTP/REST Source/Sink, Oracle CDC
Solid understanding of Kafka internals: partitioning, replication, consumer groups, offset management
Experience with Infrastructure as Code (Terraform, Ansible) for Kafka/Confluent resource provisioning
Proficiency in at least one of: Java, Python, or Go for custom connector development or Kafka Streams
Experience with CI/CD pipelines for connector deployment and configuration management
Strong troubleshooting skills for distributed systems
Nice to Have
Confluent Certified Developer or Administrator certification
Experience with ksqlDB for stream processing
AWS ecosystem experience (S3, MSK, Lambda, EventBridge)
Experience with Oracle integration (ORBC, OCI, ORDS)
Familiarity with data mesh or event-driven architecture patterns
Experience with Confluent Cloud Cluster Linking or Schema Linking
Knowledge of RBAC/ACL configuration in Confluent Platform
Experience with retail or supply chain domain data
Technical Environment
Confluent Cloud / Confluent Platform
Apache Kafka, Kafka Connect, Schema Registry, ksqlDB
AWS (S3, EC2, IAM, CloudWatch)
Terraform / Infrastructure as Code
Git, CI/CD (Jenkins, GitHub Actions, or similar)
Monitoring: Confluent Control Center, Prometheus, Grafana, Datadog
Oracle ORBC, REST APIs, ORDS (integration source systems)
Qualifications
Bachelor's degree in Computer Science, Information Technology, or related field (or equivalent experience)
Minimum 5 years overall experience in data engineering or integration roles
Minimum 3 years focused Confluent/Kafka experience
Strong communication skills — ability to work with architecture teams and business stakeholders