Senior AI Engineer – LLM Evaluation & Data Pipelines
Posted Updated
Description
Procore Technologies, Inc. in Auckland is seeking a Senior Software Engineer to own the data, tooling, and evaluation infrastructure for AI-powered features.
You will build robust data pipelines, improve model quality, and drive prompts and models from candidate selection through production rollout and validation. You will collaborate with engineering, product, and AI teams, using Python, SQL, and evaluation tools to scale AI products.
Build and maintain dataset and evaluation infrastructure with ground-truth controls. Diagnose and fix measurement-path issues between offline evals and production accuracy. Own prompt optimization from candidate selection through production rollout and post-launch validation. Automate eval pipelines and scale tooling across multiple AI products. Create reusable tooling for error analysis, audits, and experimentation. Iterate on labeling frameworks and knowledge capture systems. Analyze ontology/schema changes and their propagation across models. Strong Python skills for data manipulation, analysis, and tooling (pandas/NumPy). Solid SQL and data modeling experience with large datasets. Equity compensation Bonus incentive compensationSkills
Sign in to read and join the discussion.