Senior LLM Evaluation & Data Quality Engineer
Posted Updated
Description
Procore Technologies in Auckland is seeking a Senior Software Engineer for its Data Science team to own the accuracy of LLM-powered features that convert drone and robotics data into actionable insights.
You will collaborate with engineering, product, and AI teams to build data, tooling, and evaluation infrastructure, using Python, SQL, and LLM evaluation to improve product quality and reliability. This role reports to the Director of Engineering and is based in our Auckland office.
Build and maintain dataset and evaluation infrastructure with quality controls. Diagnose gaps between offline evals and production accuracy. Own prompt optimization across lifecycle from candidate selection to production. Automate eval pipelines and scalable tooling for multiple AI products. Create internal tooling for error analysis and experiments. Iterate on labeling frameworks and schema propagation analyses. Prioritize work by likely impact and align with teams. Strong Python and SQL experience with data manipulation and tooling. Background in data science with focus on LLM accuracy and evaluation. Experience building evaluation datasets and error analysis tools. Familiarity with ground-truth labeling and data quality checks. Knowledge of prompt engineering and ontology/schema design. Equity compensation Bonus incentiveSkills
Sign in to read and join the discussion.