AI Quality Engineer

Top 3 Reasons To Join Us

Full insurance coverage based on NET salary.

13th-month salary (paid semi-annually).

50% annual loyalty bonus (from the 2nd year).

The Job

• Define and implement quality assurance strategies for AI agents built on our bbv AI Hub product, covering functional correctness, robustness, reliability, safety, and regression behavior.
• Validate complete agent workflows, from user input and tool calls to integrations and final responses.
• Design and maintain versioned, traceable test datasets that turn business requirements and real user scenarios into test cases, including edge cases, multi-step and tool-use tasks, and safety scenarios.
• Use Langfuse to benchmark agent versions, prompts, models, and configurations against standardized datasets, tracking metrics such as task success, tool-call correctness, hallucination rate, latency, and cost.
• Design evaluation rubrics and LLM-as-a-judge prompts, combining deterministic, LLM-based, and human evaluation, and validate the reliability of automated evaluators.
• Establish baselines, quality thresholds, and regression criteria for agent releases, and support automated quality gates in CI/CD pipelines.
• Analyze failed agent traces to find root causes and turn production failures and user feedback into permanent regression test cases.
• Work closely with AI Engineers, Software Engineers, and Product Owners to define measurable acceptance criteria, resolve issues, and verify fixes.
• Report evaluation results clearly and contribute to AI agent quality standards on the bbv AI Hub.

Your Skills and Experience

• Several years of experience in Software Quality Assurance, Test Engineering, or Quality Engineering.
• Strong understanding of testing methodologies and test design techniques, with experience designing and maintaining structured test datasets.
• Experience with API and/or web application testing, automated and regression testing, and Git-based collaborative workflows.
• Strong analytical skills and the ability to trace complex failures to their root cause.
• Good understanding of AI/LLM concepts and their implications for software quality.
• Experience testing LLM-based applications, AI agents, or LLM evaluation is a strong advantage.
• Experience with some of the following is a plus: Langfuse or similar evaluation platforms, LLM-as-a-judge, LangChain/LangGraph, Python or TypeScript, CI/CD quality gates, prompt engineering, and hallucination/adversarial testing.
• Ability to communicate clearly with both technical and non-technical stakeholders.
• Good English communication skills (both speaking and writing).

Why You'll Love Working Here

• Competitive salary with full insurance coverage (medical, social, unemployment) based on net salary.
• 13th-month salary (paid semi-annually, with partial payment made mid-year rather than waiting until year-end).
• Performance review twice a year with bonus opportunities.
• Annual loyalty bonus (50% of monthly net salary from the 2nd year).
• Premium health insurance and annual health check-up.
• 14 days of leave, plus additional leave based on tenure.
• Daily perks: lunch allowance, free snacks/coffee, and sports activities.
• Learning & development support: training programs and certification sponsorship.
• English-speaking environment with tuition support.
• Team activities and events: company trips, team building, and holiday celebrations.
• Opportunity to work with international teams and join internal tech communities.