Automation Test Engineer

<div><div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>Job Title- Gen AI Tester</b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>Location- Bangalore/Hyderabad/ Chennai</b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>Roles and Responsibilities </b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>Key Responsibilities:AI QA / LLM Automation Engineer</b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>Job Overview</b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>We are seeking a highly skilled and innovative AI QA / LLM Automation Engineer to join our team. In this pivotal role, you will be responsible for designing, developing, and executing sophisticated testing strategies tailored specifically for generative AI systems. You will play a critical role in evaluating Large Language Models (LLMs), Retrieval-Augmented Generation (RAG) pipelines, and autonomous AI agents to ensure high accuracy, contextual relevance, and robust logical reasoning.</b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>Key Roles & Responsibilities</b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>AI Model Validation: Design and execute comprehensive test strategies specifically for evaluating LLM responses. Focus heavily on measuring accuracy, ensuring long-term context retention, and drastically reducing or mitigating hallucinations.</b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>AI Agent Testing: Rigorously test the underlying logic, complex state management, and tool-calling capabilities of autonomous AI agents across various edge cases.</b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>Python Automation: Develop, maintain, and scale custom automated testing frameworks from scratch using Python.</b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>CI/CD Integration: Continuously evaluate AI models by seamlessly integrating custom testing and evaluation frameworks into existing CI/CD pipelines.</b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>Evaluation Metrics: Define and implement novel evaluation metrics for non-deterministic AI features where standard deterministic software testing (exact-match) falls short.</b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>Cross-Functional Collaboration: Work closely with AI developers, data scientists, and product managers to establish quality benchmarks for AI-driven features.</b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>Required Skills & Qualifications</b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>Programming: Strong programming proficiency in Python, with an emphasis on writing clean, scalable, and maintainable automation scripts.</b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>AI/ML Knowledge: Deep understanding of Large Language Model (LLM) architectures, prompt engineering, and model behaviors.</b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>Framework Experience: Hands-on experience building or testing RAG (Retrieval-Augmented Generation) pipelines and agentic frameworks (e.g., LangChain, LlamaIndex).</b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>Non-Deterministic Testing: Proven experience in validating non-deterministic outputs where traditional exact-match assertions do not apply (e.g., semantic similarity, LLM-as-a-judge, or custom evaluation heuristics).</b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>Automation & DevOps: Solid understanding of CI/CD tools (e.g., GitHub Actions, GitLab CI, Jenkins) and version control (Git).</b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>Problem Solving: Strong analytical mindset with the ability to foresee edge cases in generative AI applications.</b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>Experience in AI evals, guardrails, and testing/automating AI agents, which are critical skills for the MAP team.</b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b> </b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>Key Responsibilities:</b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>AI Model Validation: Design and execute test strategies specifically for evaluating LLM responses, focusing on accuracy, context retention, and reducing hallucinations.</b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>AI Agent Testing: Test the logic, state management, and tool-calling capabilities of autonomous AI agents.</b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>Python Automation: Develop custom automated testing frameworks using Python to continuously evaluate AI models in CI/CD pipelines.</b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>Required Skills:</b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>Strong programming proficiency in Python.</b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>Deep understanding of LLM architectures, RAG (Retrieval-Augmented Generation), and agentic frameworks (e.g., LangChain, LlamaIndex).</b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>Experience validating non-deterministic outputs where traditional exact-match assertions do not apply.</b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>Qualifications & Required Skills- BE/ B.Tech/ Any Engineering</b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>Mandatory skills-Python, DeepEval ,AI Evals ,AIGuardrails , LLM Testing ,RAG , AI Agents , Agentic AI ,LangChain , LlamaIndex</b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>ABOUT SONATA SOFTWARE</b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b>Sonata Software is an AI-first modernization engineering company that helps enterprises transform legacy systems into intelligent, scalable business platforms. Powered by its Platformation™ framework and Harmoni.AI platform, Sonata delivers AI-led modernization across cloud, data, AI, Dynamics, test automation, and managed services. Headquartered in Bengaluru, India, Sonata has more than $1.2 billion in revenue and 6,400+ AI engineers supporting global delivery across regions including the US, UK, India, Malaysia, Mexico, Australia, DACH, and the Nordics. With deep partnerships across Microsoft, AWS, Salesforce, and Snowflake, Sonata helps Fortune 500 enterprises accelerate innovation, improve efficiency, and drive sustainable growth.</b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b><br /></b></span></font></div><div><font color="#002060" face="Calibri, Calibri_EmbeddedFont, Calibri_MSFontService, sans-serif"><span style="font-size:12px;"><b> For more information, please visit www.sonata-software.com</b></span></font></div><div><br /></div></div></div>