AI Solution Architect & Platform Lead
Posted Updated
Description
EPAM Systems in Turkey seeks an AI Solution/Principal Engineer to lead evaluation of a sovereign, multi-tenant AI platform and its applied products operating in Arabic and English. You will own how we measure and improve non-deterministic AI across assistants, retrieval, tools, and infrastructure, shaping products with trustworthy, evidence-based results.
Responsibilities include building evaluation harnesses, dataset versioning, online drift detection, and CI/CD integrations to gate quality
Define evaluation strategy: what gets measured, layers, methodology, and feed product decisions. Build evaluation harnesses, golden-set management, dataset versioning, automated grading, regression detection and reporting. Ensure grading trustworthiness through judge model selection and calibration. Assess retrieval and agents for grounding, tool-use validation, multi-step reasoning and failure recovery. Create Arabic golden sets and judges calibrated for Arabic; execute online evaluation and drift detection. Integrate evaluation and test automation as CI/CD quality gates. 5+ years building evaluation or test infrastructure and shared libraries. Leadership experience of at least 1 year. Expertise in AI evaluation spanning output quality, retrieval and grounding. Proficiency in Python; intermediate in TypeScript. Statistics knowledge for non-deterministic systems. Private health insurance Career development Mentoring programs Learning platform access English coursesSkills
Sign in to read and join the discussion.