Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

Jobgether

NewBe an early applicant

GenAI Platform Engineer – LLM Gateway

Posted Updated

Description

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a GenAI Platform Engineer – LLM Gateway based in India.

We are seeking an experienced GenAI Platform Engineer to help build, operate, and continuously improve an enterprise-scale generative AI platform. In this role, you will contribute to the development of an LLM Gateway that provides secure, scalable, and governed access to multiple large language model providers. You will design and maintain backend services, model integrations, routing capabilities, and usage management features while supporting advanced agentic AI solutions. Working within a globally distributed engineering team, you will focus on platform reliability, security, performance, and automation. You will also contribute to open-source technologies, developer enablement, and innovative AI tooling. This is an exciting individual contributor opportunity to shape the future of enterprise GenAI infrastructure in a fast-evolving technological environment.

Accountabilities:

  • Design, develop, and operate enterprise LLM Gateway features, primarily using Python, enabling secure and scalable access to multiple large language model providers.
  • Build and maintain multi-provider model integrations, intelligent routing, failover mechanisms, API key management, and access control capabilities.
  • Develop platform features for usage metering, budget management, cost transparency, and chargeback reporting.
  • Integrate agentic AI capabilities into the platform while ensuring appropriate security and governance controls.
  • Develop and maintain Model Context Protocol (MCP) servers, cross-provider tool calling, and agent-to-agent connectivity.
  • Maintain and enhance the platform's open-source foundations by assessing upstream releases, managing custom extensions, and delivering upgrades without introducing regressions.
  • Contribute to administrative interfaces, self-service portals, developer experiences, and technical documentation.
  • Build and maintain automated quality gates within CI/CD pipelines, including unit, integration, contract, and performance testing.
  • Implement observability solutions using metrics, distributed tracing, and logging tools such as OpenTelemetry and Langfuse.
  • Monitor platform performance and proactively identify and resolve latency, reliability, error-rate, and infrastructure cost issues.
  • Participate in production support and on-call rotations, leading root-cause investigations and resolving complex incidents.
  • Conduct blameless postmortems and translate incident findings into sustainable engineering improvements.
  • Apply secure-by-default engineering practices, including Single Sign-On (SSO), OAuth 2.0/OIDC, Role-Based Access Control (RBAC), and secure secrets management.
  • Maintain dependency security, version pinning, and software supply-chain hygiene.
  • Evaluate emerging generative AI technologies, including model routing solutions, guardrails, agent registries, and agentic frameworks, and provide recommendations for adoption.
  • Support internal engineering teams in onboarding and effectively using the platform through technical guides, runbooks, code samples, and reusable AI agent skills.
  • Contribute to InnerSource initiatives, participate in code reviews, and collaborate with open-source communities and technology vendors.
  • Identify opportunities to improve platform capabilities through upstream contributions, technical experimentation, and continuous innovation.

Requirements:

  • Bachelor's or Master's degree in Computer Science, Computer Applications, Engineering, or a related discipline, or equivalent practical experience.
  • At least 9 years of professional software engineering experience, including a minimum of 3 years developing and delivering production-grade GenAI or LLM-based applications and platforms.
  • Strong proficiency in Python backend development, including asynchronous programming, FastAPI or similar frameworks, REST APIs, and streaming responses.
  • Hands-on experience integrating LLM provider APIs, such as OpenAI, Azure OpenAI, and AWS Bedrock.
  • Experience implementing tool calling, function calling, and integrations across multiple LLM providers.
  • Proven experience deploying and operating services on AWS using containerization technologies and CI/CD pipelines.
  • Practical knowledge of Docker, GitHub Actions, and automated testing integrated into software delivery workflows.
  • Strong understanding of backend architecture, scalable services, API integrations, and production software engineering practices.
  • Demonstrated ability to troubleshoot complex production issues, conduct root-cause analysis, and implement long-term solutions.
  • Excellent written and verbal communication skills in English, with the ability to collaborate effectively across globally distributed teams.
  • Ability to work independently, take ownership of technical deliverables, and contribute to cross-functional engineering initiatives.

Preferred qualifications and experience:

  • Experience working with AI/LLM gateways or API gateways, particularly model routing, API key management, access controls, and cost tracking.
  • Experience operating and extending open-source software in production environments, including custom patches, upgrades, regression testing, and upstream contributions.
  • Knowledge of agentic AI architectures and tools, particularly Model Context Protocol (MCP).
  • Experience with LangGraph, LangChain, or similar agent orchestration frameworks.
  • Understanding of multi-agent orchestration, Retrieval-Augmented Generation (RAG), and vector search technologies.
  • Experience with performance and load testing, as well as observability platforms such as OpenTelemetry, Langfuse, Prometheus, and Grafana.
  • Familiarity with Kubernetes and Infrastructure as Code (IaC) tools such as Terraform.
  • Frontend development experience with React and TypeScript for building internal applications, administrative tools, and developer portals.
  • Regular use of AI-assisted development tools such as GitHub Copilot and Claude Code, with an interest in creating reusable AI agent skills.
  • Experience working in pharmaceutical, healthcare, or other regulated industries.
  • Previous experience mentoring engineers or delivering technical enablement through workshops, documentation, and onboarding programs.
  • Relevant AWS certifications are considered an advantage.

Benefits:

  • Full-time employment opportunity.
  • Fully remote working arrangement from India.
  • Opportunity to work on enterprise-scale generative AI infrastructure and LLM platform engineering.
  • Exposure to leading AI technologies and multiple LLM providers, including OpenAI, Azure OpenAI, and AWS Bedrock.
  • Hands-on experience with modern agentic AI frameworks, MCP integrations, and multi-agent architectures.
  • Opportunity to contribute to open-source projects and influence the evolution of enterprise AI platforms.
  • Professional development through exposure to cloud-native technologies, advanced automation, and emerging AI capabilities.
  • Collaboration with globally distributed engineering teams and international technical specialists.
  • Opportunity to take ownership of impactful engineering projects and influence platform architecture and technical direction.
  • Regular collaboration with colleagues in European time zones.
  • Participation in a structured production support and on-call rotation.
How Jobgether works:
We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.
We appreciate your interest and wish you the best!
Why Apply Through Jobgether?
Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.
#LI-CL1

Skills

Apply

See also

AI Engineering jobs by country — openings, pay and top skills →