AI/ML Quality Assurance Engineer
Posted Updated
Description
Robots & Pencils in Colombia is seeking a Testing Engineer to build and maintain quality across software features and AI/ML systems. You’ll own automated tests, validate model outputs, and integrate QA into delivery, working with a cross-functional team to ship reliable AI-enabled software.
You’ll contribute to quality strategy, participate in sprint ceremonies, and help scale testing across UI, API, and ML pipelines while advancing best practices in CI/CD and data validation.
Write and maintain automated test scripts for UI, API, and integration testing. Validate ML model outputs including accuracy, consistency, and expected behavior. Test data pipelines and AI workflows, performing dataset validation and data quality checks. Execute automated test suites, analyze results, document and track defects. Integrate automated tests into CI/CD pipelines to support continuous quality. Support model evaluation processes and assist in testing AI workflows end-to-end. Test AI/ML systems, including LLM evaluation and safety checks. 2+ years of experience in test automation or AI/ML testing. Proficiency in Python, JavaScript, Java, or similar, with the ability to write and maintain test code independently. Experience with test automation frameworks for UI, API, or ML workflows. Understanding of AI/ML concepts and the ML lifecycle, with the ability to validate model outputs against defined criteria. Experience with CI/CD pipelines and version control. Working knowledge of API testing and data validation. Solid understanding of SDLC/STLC and Agile methodologies. Proficiency with bug-tracking and test management tools. Familiarity with testing AI/ML systems, including LLM evaluation, RAG retrieval quality, model drift monitoring, and AI safety considerations.Skills
Sign in to read and join the discussion.