AI Automation Engineer OR LLM Automation Engineer
Description
AI/LLM Test Automation Engineer (SDET)
Location: Bangalore (Hybrid preferred)
Experience: 5-8 Years
Job Summary
We are seeking a Senior Test Automation Engineer (SDET) with expertise in AI/LLM testing and Azure DevOps CI/CD to build robust automation frameworks for cutting-edge AI applications. The role combines deep programming skills (Java/Python), modern DevOps practices, and specialized LLM testing to ensure high-quality AI product delivery.
Key Responsibilities
- Design, develop, and maintain automation frameworks using Java/Python for web, mobile, API, and backend testing.
- Create and manage YAML-based CI/CD pipelines in Azure DevOps for end-to-end testing workflows.
- Perform AI/LLM testing including prompt validation, content generation evaluation, model behavior analysis, and bias detection.
- Write and maintain BDD Cucumber feature files integrated with automation suites.
- Execute manual + automated testing across diverse application layers with focus on edge cases.
- Implement Git branching strategies, code reviews, and repository best practices.
- Track defects and manage test lifecycle using ServiceNow or similar tools.
- Conduct root-cause analysis, troubleshoot complex issues, and drive continuous quality improvements.
Mandatory Skills & Experience
✅ 5+ years SDET/Automation experience
✅ Java/Python scripting for test frameworks (Selenium, REST Assured, Playwright)
✅ Azure DevOps YAML pipelines (CI/CD end-to-end)
✅ AI/LLM testing (prompt engineering, model validation, RAG testing)
✅ Cucumber BDD (Gherkin feature files + step definitions)
✅ Git (branching, PRs, GitFlow)
✅ ServiceNow/Jira defect tracking
✅ Manual + Automation testing (web/mobile/API/backend)
Technical Stack
Programming: Java, Python, JavaScript
CI/CD: Azure DevOps, YAML Pipelines
Testing: Selenium, Playwright, REST Assured, Postman
BDD: Cucumber (Gherkin), JBehave
AI/ML: Prompt validation, LLM APIs (OpenAI, LangChain)
Version Control: Git, GitHub/GitLabDefect
Tracking: ServiceNow, Jira, Azure Boards
Preferred Qualifications
- Exposure to AI testing frameworks (LangSmith, Promptfoo)
- Experience with containerization (Docker) and Kubernetes
- Knowledge of performance testing for AI workloads
- AWS/GCP cloud testing experience
- ISTQB or relevant QA certifications
What We Offer
- Work on next-gen AI/LLM products with global impact
- Modern tech stack with Azure-native DevOps
- Flexible hybrid/remote work model
- Continuous learning opportunities in AI testing
Skills
- AI
- API
- Automation
- AWS
- Azure
- Azure DevOps
- BDD
- CI/CD
- Cloud
- Containerization
- Cucumber
- DevOps
- Docker
- GCP
- Git
- Gitflow
- GitHub
- Java
- JavaScript
- Jira
- Kubernetes
- LangChain
- LangSmith
- LLM
- Machine Learning
- OpenAI
- Playwright
- Postman
- Prompt Engineering
- Python
- RAG
- REST Assured
- Root Cause Analysis
- Selenium
- ServiceNow
- Test Automation
- Version Control
- YAML
Sign in to read and join the discussion.