Kafka Architect / Streaming Data Architect

Job Title – Kafka Architect / Streaming Data Architect

Company – TCS (MEA)

Job type – Full time

About Us

Tata Consultancy Services (TCS) is an IT services, consulting and business solutions organization that has been partnering with many of the world’s largest businesses in their transformation journeys for over 50 years. TCS offers a consulting-led, cognitive powered, integrated portfolio of business, technology and engineering services and solutions. This is delivered through its unique Location Independent Agile™ delivery model, recognized as a benchmark of excellence in software development.

A part of the Tata group, India's largest multinational business group, TCS has over 616,171 of the world’s best-trained consultants with 157 nationalities in 53 countries.

JOB DESCRIPTION

KEY RESPONSIBILITIES

• Design enterprise-scale Kafka architecture aligned with Bank Lakehouse (Medallion - Bronze/Silver/Gold).
• Define event-driven architecture patterns for real-time ingestion and processing.
• Establish topic strategy, partitioning, replication, retention policies.
• Architect multi-cluster Kafka setup (dev/test/prod, DR strategy).
• Ensure exact point-in-time recovery and fault tolerance.

PLATFORM IMPLEMENTATION

• Lead setup and configuration of:
• Kafka Connect framework
• Schema Registry
• Integrate Kafka with:
• Cloudera Data Platform (CDP)
• Data Warehouse (Teradata)
• Enable ingestion from source systems.
• Support incremental and historical loads integration.

PERFORMANCE & SCALABILITY

• Ensure:
• High throughput and low latency
• Support for concurrent consumers without degradation
• Optimize:
• Consumer group scaling
• Producer configurations (acks, batching, compression)

RESILIENCE & RELIABILITY

• Implement:
• Failover strategies
• Data replication and durability (99.9% uptime requirement)
• Handle:
• Hardware failures
• Design retry, dead-letter queues (DLQ), error handling frameworks

DATA GOVERNANCE & SECURITY

• Enforce:
• Role-Based Access Control (RBAC)
• Encryption (in transit & at rest)
• Data masking/tokenization support (aligned with classification)
• Integrate with:
• Metadata and lineage tools
• Governance frameworks (NDMO/NDI compliance)

MONITORING & OPERATIONS

• Implement monitoring using:
• Cloudera Manager
• Kafka metrics (JMX, Prometheus, Grafana)
• Define:
• Alerting thresholds
• Ensure observability for:
• Throughput and error rates

PHASE 1 REMEDIATION SUPPORT

• Assess existing Kafka pipelines and components.
• Identify:
• Redesign/rebuild where required.
• Ensure alignment with target architecture and data models (FSLDM/FSAS).

REPORTING ENABLEMENT SUPPORT

• Ensure:
• Real-time data availability for downstream reporting.
• Support:
• Query performance optimization
• Data consistency across Bronze/Silver/Gold

DOCUMENTATION & GOVERNANCE

• Deliver:
• Topic catalog and schema definitions
• Maintain:
• Source-to-target mappings
• Architecture governance boards

KEY TECHNICAL SKILLS

CORE TECHNOLOGIES

• Cloudera Kafka (mandatory)
• Kafka Connect, Kafka Streams
• Schema Registry (Avro/Protobuf/JSON)

ECOSYSTEM TOOLS

• Cloudera (CDP)
• Apache Flink (preferred)

DATA & INTEGRATION

• REST APIs, Microservices integration
• ETL/ELT Design patterns

DEVOPS & INFRA

• Kubernetes / Docker (optional but preferred)

BANKING DOMAIN EXPERIENCE (MANDATORY)

• Experience in BFSI domain
• Understanding of regulatory reporting

SOFT SKILLS

• Strong stakeholder management (business + IT)
• Ability to work in multi-vendor environment
• Experience in large-scale transformation programs
• Strong problem-solving and RCA skills

Application Deadline: 30- July -2026