AI Engineer (Trust & Evaluation)
Posted Updated
Description
- Designs, develops and deploys the evaluation and trust capabilities of Equinix’s GenAI platform
- Builds platform features that let teams evaluate their agents at scale: golden datasets, synthetic test data, scoring with LLM-as-judge, and monitoring of agent quality in production
- Builds and tests the trust controls that agents run on, including agent identity, access, guardrails and data boundaries
- Traces agent failures to their root cause and fixes them in the platform
- Works in a team with senior engineers and with product, platform and security partners to turn defined problems into robust, secure, observable and performant solutions
- Applies solid knowledge of software engineering practice and of LLM and agent concepts (prompting, retrieval, tool and function calling, orchestration, MCP)
- Understands evaluation methods (code-based checks, LLM-as-judge, human review, golden datasets, regression suites) and evaluation statistics (sampling, variance across runs, agreement between reviewers)
- Understands AI security risks such as prompt injection and data leakage, and identity concepts such as OAuth and per-user access
- Follows established standards and suggests improvements
- Works on problems of defined scope where analysis requires evaluation of identifiable factors
- Traces an agent failure to its cause (retrieval, tool call, prompt, model, data or infrastructure) using logs and traces, and follows it until it is fixed
- Judges whether a metric reflects real quality
- Escalates unusual or high-impact issues
- Work is reviewed at defined checkpoints
- Works with AI engineers, product managers, forward-deployed engineers, platform and security teams
- Turns the needs of agent builders into platform features
- Explains results, defects and risks to partners who are not familiar with the subject
- Asks for guidance early and shares knowledge with the team
- Build platform features for evaluation: dataset creation and versioning, golden datasets, synthetic test data and reusable metric templates
- Build scoring for agents: code-based checks, LLM-as-judge, tool-call accuracy and grounding checks
- Build human-review features that compare judge scores against human reviewers, and recalibrate when agreement is low
- Run evaluations at scale: before release, on every change to a prompt, model or tool, and on agents in production
- Build evaluation playbooks so agent builders can evaluate their agents without specialist help
- Keep results traceable to the agent, model, prompt and dataset versions tested, with a record of who approved each release
- Build and test the platform’s trust controls: agent identity, per-user access to connected systems, tool permissions, guardrails and data boundaries
- Run adversarial tests and red-teaming (prompt injection, out-of-scope requests, data-boundary tests), and turn findings into fixes and regression tests
- Work with security and governance teams to turn policy requirements into automated checks
- Investigate agent failures and performance problems end to end, and follow them until the cause is fixed
- Build test and performance tooling: end-to-end agent tests, load and latency tests, and failure diagnostics
- Turn production failures and user feedback into regression tests, so the same defect does not return
- Write tested, reviewed and documented code, and contribute to shared libraries and tooling
- Test and build to security, data-privacy and responsible-AI requirements
- Bachelor’s degree in computer science, engineering, data science, mathematics or a related field, or equivalent experience
- Ability to write SQL and analyze results
- Working knowledge of LLM applications (prompting, retrieval, agents)
- Hands-on experience building, not only using, evaluations or test systems for ML or LLM applications, including reading failures from logs or traces
- 1–4 years in AI/ML or software engineering (internships and applied research count)
- Working knowledge of evaluation statistics (sampling, confidence intervals, variance across runs) and of measuring agreement between reviewers
- Clear written and verbal communication
- Proficiency in Python, with solid habits: version control, testing, code review. Experience building backend services or platform features
- Experience building evaluation or observability features used by many teams or at scale
- Experience with LLM evaluation and observability tools, and tracing standards such as OpenTelemetry
- Experience with red-teaming or adversarial testing of LLM systems
- Experience with identity and access (OAuth 2.0, OIDC, delegated access)
- Experience with performance, load or test engineering
- Experience with agent frameworks, MCP or A2A
- Experience with Google Cloud (Vertex AI, GKE) and Kubernetes
- Familiarity with AI risk frameworks (NIST AI RMF, ISO 42001)
- MS in a related field
Skills
Sign in to read and join the discussion.