Senior Python Engineer — AI Coding Evaluation Architect
Summary
Designs and builds evaluation datasets for AI coding agents at Mindrift: crafting realistic simulated dev environments (codebase, tickets, docs), creating tasks from intermediate states, and writing tests that accept valid solutions and reject incorrect ones, with iteration based on QA. Core tech: Python.
Mindrift is building a dataset to evaluate AI coding agents—assessing how well a model handles real‑world developer tasks. You will craft challenging tasks within realistic simulated environments: a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history.
The work demands designing tasks from intermediate states, writing tests that accept all valid solutions and reject incorrect ones, and iterating on tasks based on QA