Senior AI Test and Automation Engineer
Posted Updated 116
views
Description
Senior AI Test and Automation Engineer
Company: Cadence Design Systems India Pvt Ltd
Job Location: Bengaluru, Karnataka, India
Employment Type: Full-time
Experience Required: 7+ years
Principal Software Engineer
At Cadence, we hire and develop leaders and innovators who want to make an impact on the world of technology.
Senior AI Test / Automation Engineer Overview
Role: AI Test / Automation Engineer
Location: Bangalore India
Department: AI Engineering / Quality Assurance
Experience Level: Mid to Senior
About the Role
We are looking for a highly motivated Senior AI Test / Automation Engineer to design and scale automated validation frameworks for AI/ML models, LLM-based applications, and agentic systems . This role is critical to ensure that AI solutions meet enterprise standards for quality, reliability, safety, and compliance before and after production deployment.
Key Responsibilities
Build and maintain AI test automation frameworks for pre-qualification and continuous validation of models and agent workflows
Develop comprehensive test suites , including: Unit, integration, and end-to-end (E2E) Functional, regression, performance, and safety testing
Validate AI system behavior , including: Non-deterministic LLM outputs Hallucinations and edge cases Multi-step agent decision-making
Design and manage evaluation systems : Golden datasets Benchmarking pipelines (accuracy, latency, reliability)
Automate testing within CI/CD pipelines for model updates, prompt changes, and tool integrations
Implement observability and telemetry to enable traceability, monitoring, and audit readiness
Collaborate cross-functionally with ML, MLOps, Product, and Security teams to define quality gates and release criteria
Track and report quality KPIs , including test coverage, defect leakage, and system reliability
Drive root-cause analysis and continuous improvement across the AI testing lifecycle
Required Skills
Core Engineering
Strong programming skills in Python ; familiarity with Bash, TypeScript, or Go
Experience with test automation frameworks such as PyTest, Playwright, Selenium, or Cypress
Proficiency in CI/CD tools (GitHub Actions, Jenkins, GitLab CI)
Experience with cloud platforms (AWS, Azure, GCP) and containers (Docker, Kubernetes)
AI / ML & Agentic Systems
Hands-on experience with LLM ecosystems (OpenAI, Anthropic, Bedrock)
Familiarity with: RAG architectures and vector databases (Pinecone, Weaviate)
Agent frameworks (Lang Chain, Llama Index, Auto Gen)
AI Testing Techniques
Experience with non-deterministic testing approaches (statistical assertions, tolerance thresholds)
Knowledge of evaluation methods : LLM-as-a-judge BLEU, ROUGE, semantic similarity scoring
Experience with prompt and agent regression testing
Understanding of AI safety testing , including adversarial testing, bias/fairness validation, and jailbreak detection
Tooling (Preferred)
AI testing & observability tools: Lang Smith, TruLens, Arize, Weights & Biases
Evaluation tools: Deep Eval, Ragas, Prompt Foo, Giskard
Monitoring: Prometheus, Grafana, Open Telemetry
Soft Skills
Strong analytical and problem-solving skills
Excellent communication and cross-functional collaboration
Data-driven mindset with focus on quality KPIs
Detail-oriented with a strong bias toward automation and scalability
Experience Requirements
7+ years in QA, SDET, or test automation engineering
Proven experience building and scaling automation frameworks
Hands-on experience with AI/ML systems or LLM-based applications
Experience testing RAG pipelines or agentic workflows
Owned end-to-end AI test strategy and architecture
Defined quality metrics and release gates
Delivered scalable validation pipelines for production AI systems
Supported audit and compliance readiness
Preferred
Experience in enterprise or regulated environments (SOC2, ISO 27001, etc.)
Exposure to: Shift-left testing practices Production observability and monitoring Chaos or resilience testing
Senior-Level Differentiators
Education Bachelors or Masters degree in Computer Science, Software Engineering, or related field
Nice-to-have: ISTQB certification Cloud/ML certifications (AWS, Azure, GCP) AI testing certifications
What Success Looks Like
AI systems that are accurate, reliable, and safe
Fully automated test pipelines integrated into CI/CD
Measurable improvements in defect leakage and model quality
Strong observability and auditability across AI systems
Scalable validation frameworks supporting rapid AI innovation
Were doing work that matters. Help us solve what others cant.
Experience Level Senior Level .
Skills
- Agentic AI
- AI
- Anthropic
- Automation
- AWS
- Azure
- Bash
- CI/CD
- Cloud
- Cypress
- Docker
- GCP
- GitHub
- GitHub Actions
- GitLab
- Grafana
- ISO 27001
- Jenkins
- Kubernetes
- LlamaIndex
- LLM
- Machine Learning
- MLOps
- Observability
- OpenAI
- OpenTelemetry
- Pinecone
- Playwright
- Prometheus
- pytest
- Python
- Root Cause Analysis
- Selenium
- SOC 2
- Test Automation
- TypeScript
- Vector Databases
- Weaviate
Sign in to read and join the discussion.