Agent Evaluation Engineer
Posted Updated
Description
Employment seeks experienced software developers and testers to help build a realistic developer environment and evaluation tasks for AI coding agents. You will craft prompts, define success criteria, and write tests that accept multiple valid approaches.
This project uses Python/FastAPI, TypeScript/React, Docker, PostgreSQL, Kafka and Redis; collaboration with a remote team is required and English proficiency (B2+) is essential.
Create realistic developer environments including codebase, infra, tickets and docs. Design tasks from intermediate states and define what ‘solved’ means. Write tests that accept multiple valid approaches and reject incorrect ones. Iterate on tasks based on QA feedback to ensure fairness and robustness. 5+ years of professional software development experience. Strong hands-on with Python (FastAPI) and JS/TS (React). Experience with Docker, PostgreSQL, Kafka and Redis. Proven testing experience (functional/integration tests) and English proficiency (B2+).Skills
Sign in to read and join the discussion.