Lead Backend & Data Engineer – AI Platform

At Avenga, we believe that human creativity empowers technology that matters. Operating globally, our 6000+ specialists provide a full spectrum of services, including business and tech advisory, enterprise solutions, CX, UX and Ul design, managed services, product development, and software development.

This is the job

We're looking for a Lead Backend & Data Engineer to help build the foundation of a next-generation, deterministic Retrieval-Augmented Generation (RAG) system. Unlike standard semantic RAG architectures, our system fuses vector search with Google Cloud Spanner's property graph capabilities. You'll build the data ingestion pipelines and high-concurrency API layer that serve as the 'Truth Engine' for our downstream LLM agents.

This is you

• 7+ years of experience in Backend Engineering, Data Engineering, or Distributed Systems
• Deep expertise in Apache Beam (Python) or equivalent large-scale distributed data processing frameworks, with the ability to build production-grade Apache Beam pipelines.
• Hands‑on production experience with Google Cloud Spanner or another distributed SQL database (e.g. CockroachDB, YugabyteDB, TiDB).
• Working knowledge of graph databases or graph query languages such as Cypher, Gremlin, Neo4j, or ISO GQL.
• Advanced experience building production APIs using asynchronous Python (FastAPI) and Pydantic.
• Hands‑on production experience building and supporting workflow orchestration using Temporal / Cadence or a similar workflow orchestration platform.
• Deep understanding of separating unpredictable operations into Activities while keeping Workflows strictly deterministic.
• Proven ability to implement automated retries, compensation logic, and state management in distributed systems.
• Strong analytical, architectural, and problem‑solving skills.
• Upper‑Intermediate (B2) or higher level of English.

Nice to have

• Experience with LangChain and/or LangGraph
• Familiarity with Retrieval-Augmented Generation (RAG) architectures
• Experience with Google Cloud Dataflow and Flink
• Experience with distributed data processing frameworks such as Spark or Flink
• Experience with vector databases
• Degree in Computer Science, Software Engineering, or a related field

This is your role

• Design and deploy Apache Beam pipelines capable of executing on varied runners (Dataflow, Flink)
• Structure and optimize multi‑hop graph queries using ISO GQL to trace physical and logical provenance
• Implement advanced gRPC connection pooling for the Spanner client within a FastAPI/Cloud Run environment to ensure sub‑100ms database latency
• Define strict API data contracts using Pydantic for seamless integration with downstream LangChain/LangGraph agent payloads
• Design a FastAPI architecture that delegates short‑lived queries to synchronous I/O handlers and offloads complex, multi‑hop scenario processing to Temporal
• Write and orchestrate Workflow Definitions and Activity Definitions, and configure Temporal Clients using the Temporal Python SDK
• Deploy and manage a horizontally scalable fleet of Temporal Worker processes executing heavy database queries and AI routing logic
• Collaborate closely with AI, Platform, and Data Engineering teams to deliver production‑ready solutions
• Contribute to system architecture, technical decision‑making, code reviews, and engineering best practices

What awaits you at Avenga?

At Avenga, everyone matters. We provide equal opportunities in recruitment, career development, and leadership, regardless of race, ethnicity, gender identity, sexual orientation, disability, age, religion, or any other characteristic. We are committed to fostering a work environment where our diverse community of employees, candidates, and business partners actively shapes our growth. By bringing together people from different backgrounds and experiences, we build a workplace where everyone feels free to be themselves while honoring the boundaries of others.