Principal AI Data & Evaluation Strategist
You will serve as a consultative expert and thought partner alongside program managers, providing strategic guidance during project discovery, solution design, data planning, evaluation-framework creation, quality-rubric development, and pilot execution. You'll work directly with stakeholders, engineering teams, researchers, and program leadership to define project requirements before execution begins. This is not an operational-delivery role; you'll bring subject-matter expertise, challenge assumptions, identify risks early, and ensure projects are designed for quality, scalability, and measurable outcomes. It is a remote, flexible engagement contributing to frontier AI work.
Responsibilities
- Lead project discovery and translate business goals into AI data requirements
- Define project success metrics and identify risks and failure modes early
- Design training and evaluation datasets and define taxonomy structures
- Recommend data-sourcing methodologies and establish ground-truth standards
- Identify coverage gaps and edge cases
- Develop human-evaluation frameworks and quality-measurement methodologies
- Establish acceptance criteria and design benchmarking approaches
- Build calibration mechanisms
- Challenge assumptions that may impact project quality
- Recommend industry best practices and guide teams through ambiguity
- Support executive and client reviews
Requirements
- 10+ years in AI, data science, machine learning, research, or AI operations
- Experience designing large-scale AI data programs
- Experience with LLM evaluation, human feedback systems, AI benchmarking, and dataset development
- Strong written English and the ability to communicate complex ideas clearly
- Comfortable working independently in a remote environment
- Based in India