Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

Workday

NewBe an early applicant

Senior AI Evaluation Engineer (CICD & Platform)

Posted Updated

Description

  • The Forward Intelligence team owns the skills that power Workday’s deployment agent—turning the deep configuration expertise and hard-earned wisdom of our Domain Experts and Deployment Consultants into production-grade, fully tested skills that the agent can execute lawfully in live customer tenants
  • What we own: The end-to-end skill engine and governance ecosystem, spanning skill knowledge extraction, promotion pipelines, evaluation frameworks, self-correcting drift detection, and developer authoring environments
  • How we build: Automated testing harnesses and release-gating pipelines running at enterprise scale, ensuring deterministic execution, strict schema compliance, and zero regressions for skills executing in live customer tenants
  • Agentic engineering: We build evaluation tooling, synthetic datasets, and automated red-teaming pipelines that help engineers test how AI agents and Model Context Protocol (MCP) skills perform across complex tenant environments
  • How we work: Close end-to-end collaboration across domain experts, AI platform engineers, and site reliability teams to transform domain knowledge into trustworthy, production-ready AI skills
  • We are seeking a Senior AI Evaluation Engineer (CI/CD & Platform) to own the continuous evaluation architecture, deployment pipeline infrastructure, and release gating for our enterprise AI Skills platform
  • In this role, you will build both the MLOps/CI-CD deployment pipelines and the automated eval harnesses that ensure all production AI skills execute accurately, remain schema-compliant, and run safely in live customer environments
  • You will bridge the gap between AI skill development and production platform engineering—leveraging modern evaluation frameworks (e.g., LangSmith, Braintrust, DeepEval) alongside CI/CD pipelines to evaluate agent trajectories, test schema contracts, and certify every skill before deployment
  • Continuous Evaluation:. Build tiered evaluation pipelines—separating instant static/schema checks for PR developer loops from comprehensive synthetic, multi-turn trajectory, and LLM-as-a-Judge eval suites for release promotion gates
  • CI/CD Quality Gates & MLOps Infrastructure: Architect and maintain CI/CD build pipelines (GitHub Actions, GitLab CI), skill registry versioning, and runner environments to enforce automated “pass/fail” release gates on every pull request
  • Deterministic Code & Schema Validation:. Develop synthetic data generation pipelines and automated red-teaming utilities that allow skill authors to instantly generate edge-case evaluation suites from raw API schemas and domain specifications
  • Benchmark Datasets & Regression Testing: Curate versioned golden datasets and execute continuous regression testing across base model upgrades, prompt changes, and security boundary checks
  • Production Observability: Ingest real-world production traces and failure logs back into benchmark datasets to continuously close test coverage gaps
  • 1+ years of hands-on experience evaluating or building Generative AI applications (LLM tool-calling, agent workflows, or structured output parsing)
  • AI Evaluation Tooling: Direct experience with modern evaluation and observability frameworks (e.g., LangSmith, Braintrust, DeepEval, Promptfoo, Phoenix)
  • AI Security & Governance: Understanding of core LLM failure modes (hallucinations, prompt injection, PII/PHI sanitization, RBAC boundaries)
  • 4+ years of professional software engineering experience in Python building production-grade systems—with hands-on strength in EITHER CI/CD pipeline engineering OR AI evaluation methodology, plus working familiarity with the other
  • Static Code Analysis: Experience using schema validation frameworks (Pydantic, JSON Schema), linters, and test automation tooling to enforce agreed code contracts
  • Technical Communication & Leadership: Demonstrated ability to own problem domains end-to-end, collaborate closely with platform and product teams, and write clear technical documentation and architecture designs
  • Agentic Standards & MCP: Familiarity with the Model Context Protocol (MCP), tool-calling specifications, or agent execution harnesses
  • Schema Contracts & API Validation: Strong knowledge of schema validation libraries (JSON Schema, Pydantic, OpenAPI) and API contract testing
  • Bachelor’s degree in Computer Science, Engineering, or a related field, or equivalent practical experience
  • 2+ years designing, building, and operating test automation frameworks (pytest), CI/CD build pipelines (GitHub Actions, GitLab CI), or automated release gates
  • Quality & Testing Discipline: Strong engineering rigor around test coverage, dataset versioning, evaluator calibration metrics, and regression prevention in continuous deployment pipelines

Skills

Apply

See also