Member of Technical Staff - Data Quality Engineer Post-training
Posted 3
views
Summary
Reflection AI is hiring a Data Quality Engineer for its post-training team. Day to day, this person defines measurable data-quality standards for LLM training and evaluation, builds automated QA pipelines in Python over large datasets, gives feedback to external data vendors, and monitors quality trends to improve acceptance criteria.
You will define and operationalize measurable data-quality standards for LLM post-training and evaluation. You will analyze datasets, build and validate automated QA methods, create reusable quality pipelines, provide feedback to vendors, and monitor quality trends to improve acceptance criteria.
Responsibilities
- Own upstream data quality for LLM post-training and evaluation
- Translate requirements into measurable quality signals
- Provide actionable feedback to external data vendors
- Design, validate, and scale automated QA methods
- Build reusable QA pipelines for model training and evaluation
- Monitor and report data quality over time
- Improve quality standards, processes, and acceptance criteria
Requirements
- Experience building data pipelines, QA systems, or evaluation workflows for post-training data and agentic environments
- Understanding of SFT, RL, training, and evaluation
- Experience designing automated quality checks
- Proficiency in Python
- Experience building ML or LLM workflows
- Experience with large datasets and automated evaluation or quality-checking systems
- Familiarity with LLM training and evaluation
- Communication skills
Benefits
- Stock options
- Medical, dental, vision, and life insurance
- Annual wellness allowance
- Daily office lunch and dinner
- 22 weeks of paid parental leave
- Unlimited paid time off in the U.S.
- 30 vacation days in the U.K.
- Visa sponsorship support
- Regular off-sites, happy hours, and team celebrations
