AI Engineer with Deep Data Science Understanding
Summary
Builds and evaluates reliable LLM/agent components for an AI Skills Registry, running evaluation frameworks (error analysis, rubric design, LLM-as-judge calibration) and tracking quality and cost metrics for production agents. Core stack: Python, AWS cloud ML infrastructure (Bedrock, AgentCore), Arize AX.
Summary: The primary goal of the AI Engineer role is to contribute to the development of the AI Skills Registry, focusing on building reliable components for AI solutions that operate effectively across various platforms.
Responsibilities:
Build components for the AgentCore and/or Amazon Quick and run evaluation frameworks for skills and tools builds, covering error analysis, rubric design, and LLM-as-judge calibration.
Ensure the reliability and accuracy of the AI components developed.
Help operationalize the Evaluate and Iterate phase, ensuring evaluation practices are robust across agent builds.
Define and track quality and cost-per-outcome metrics for production agents, using tools such as Arize AX.
Run structured error analysis on agent outputs to identify failure patterns and implement fixes.
Key Requirements:
Hands-on experience evaluating ML or LLM-based systems in production.
Working knowledge of evaluation methodology, including error analysis and rubric design.
Strong Python and data analysis skills.
Solid experience with cloud-native ML infrastructure (e.g., AWS Bedrock).
Ability to work independently in a developing team environment.
Nice to Have:
Experience with AWS AgentCore, Strands Agents, or similar agent runtimes.
Experience in a regulated industry (financial services or healthcare) and familiarity with GDPR or equivalent data protection regimes.
Other Details:
Location: On-site or remote work options are available.
Start/End: ASAP to 31/12-26 with an option for extension of 3-12 months.