Member of Technical Staff Data Flywheel
Posted 5
views
Summary
Member of Technical Staff on Reflection's Data Flywheel team (New York, London, or San Francisco) who sources high-value data, designs evaluations, graders, and reward signals, analyzes LLM failure modes, and builds data pipelines that turn human and synthetic data programs into measurable model improvements.
You will identify and integrate data sources, design evaluations and feedback loops, analyze model failures, create targeted datasets and reward signals, run human and synthetic data programs, build reliable data pipelines, and turn new signals into measurable model improvements.
Responsibilities
- Identify high-value data sources and partnership opportunities
- Translate use cases into representative evaluations
- Bring data sources into production evaluation and training pipelines
- Design evaluations, graders, and feedback loops
- Analyze model performance and failure modes
- Create targeted datasets, reward signals, and training interventions
- Develop human and synthetic data strategies
- Run evaluation and data-collection programs with vendors
- Build infrastructure for data ingestion, inspection, versioning, and evaluation
- Collaborate to turn new signals into measurable model improvements
Requirements
- Computer science or machine learning knowledge
- LLM training and evaluation
- Evaluation design
- Data curation
- Reinforcement learning
- Reward design
- Software engineering
- Automated data pipelines
- Large-scale machine learning systems
- End-to-end project ownership
Benefits
- Stock options
- Medical, dental, vision, and life insurance
- Annual wellness allowance
- Daily office lunch and dinner
- 22 weeks of paid parental leave
- Unlimited paid time off in the U.S.
- 30 vacation days in the U.K.
- Visa sponsorship support
- Regular off-sites, happy hours, and team celebrations
