AI Developer (LLM Evaluation)
Summary
An AI developer designs and builds an automated evaluation and regression-testing framework for a deployed LLM assistant running on AWS Bedrock, integrating test runs into CI/CD with support from an internal DevOps team. Core stack: LLM eval tools (Promptfoo, Langfuse, Arize), AWS, CI/CD, with Kotlin/Spring as a plus.
IT Craft is looking for an experienced AI Developer with hands-on expertise in LLM evaluation and automated testing to join our team. In this role, you will take ownership of designing and implementing an evaluation framework for a deployed AI assistant running on AWS Bedrock, with infrastructure support from the internal DevOps team.
Key Responsibilities:
Design and implement automated end-to-end and regression testing infrastructure for an LLM-based assistant;
Build an evaluation harness that runs a structured set of prompt scenarios against the deployed service;
Implement automated checks for response correctness, tool usage, and guardrail behavior;
Generate clear pass/fail results based on agreed evaluation baselines;
Collaborate with the internal DevOps team to integrate evaluation runs into CI/CD processes and identify quality and behavior regressions before and after deployments;
Analyze evaluation results and continuously improve test coverage and reliability;
Plan and organize the implementation independently, define follow-up tasks, and coordinate with internal specialists through regular synchronization meetings.
Required Skills:
Hands-on experience with LLM evaluation tools such as Promptfoo, Langfuse, Arize, or comparable solutions;
Practical experience designing or implementing automated evaluation and regression testing for LLM-based applications;
Understanding of approaches used to evaluate non-deterministic LLM responses, tool calls, and guardrail behavior;
Solid experience with AWS and CI/CD;
Ability to design a solution from scratch and deliver it independently;
Familiarity with Kotlin and Spring would be a plus;
Strong written and spoken English (Upper-Intermediate+ / Advanced).
IT Craft offers:
Competitive compensation according to the qualifications;
Flexible working hours, remote work;
Opportunity for career growth;
Compensation for technical courses, educational resources, and sport;
In-house English training;
A friendly team of open-minded people.
Please send your CV.