Senior/Lead Research Engineer (Foundation Models)
Description
This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Senior/Lead Research Engineer (Foundation Models) based in the United States.
We are looking for a Senior/Lead Research Engineer to build and scale the infrastructure behind advanced foundation models for high-stakes professional workflows. You will work closely with research scientists and engineering teams to develop reliable systems for model training, evaluation, and experimentation. The role offers two areas of focus: large-scale reinforcement learning (RL) training for agentic Mixture-of-Experts models, or evaluation engineering for large language models (LLMs). You will tackle complex distributed computing and performance challenges while optimizing GPU utilization, cost, and latency. Your work will help accelerate model development, improve release quality, and support the reliable deployment of AI capabilities. At Lead level, you will also shape technical architecture, guide delivery, and mentor other engineers.
Accountabilities
-
Research and Engineering Infrastructure: Build, operate, and maintain the systems used to train, evaluate, and experiment with foundation models, ensuring reliability as model complexity and adoption increase.
-
Large-Scale RL Training: For candidates focused on training infrastructure, architect and optimize systems supporting reinforcement learning for agentic Mixture-of-Experts models exceeding 300 billion parameters, including asynchronous and long-context training with tool interaction.
-
Evaluation Engineering: For candidates focused on evaluation, build, scale, and optimize evaluation suites and leaderboards that support model releases and product integrations.
-
Performance Optimization: Profile and improve systems for cost efficiency, latency, and GPU utilization without compromising model quality. Document technical trade-offs and optimization decisions for the wider team.
-
Complex Technical Troubleshooting: Investigate and resolve challenging numerical, distributed systems, and performance issues across training frameworks, inference engines, and evaluation code.
-
Research Collaboration: Work directly with research and evaluation teams to translate new algorithms, reward mechanisms, and evaluation approaches into scalable systems that can be tested quickly and reliably.
-
Model Release Support: Collaborate with training and evaluation teams to prepare new model releases, complete testing, and support deployment into production environments.
-
Production Readiness and Compliance: Strengthen production engineering practices and contribute to the organization's ISO 27001 compliance journey.
-
Engineering Standards: Establish and promote high standards for code quality, testing, reproducibility, and maintainability.
-
Technical Leadership (Lead Level): Own the architecture of your technical area, coordinate delivery plans and staffing with research leads, and mentor engineers while driving complex projects from conception through completion.
Requirements:
-
Education: Master's or PhD in Computer Science, Machine Learning, or a related discipline, or equivalent professional industry experience.
-
Software Engineering: Strong software engineering fundamentals, including experience with large-scale distributed systems and the ability to work effectively in substantial open-source codebases.
-
Production and Research Systems: Experience maintaining reliable research or production systems under real-world workloads, including debugging distributed systems and resolving operational incidents.
-
Research Collaboration: Demonstrated ability to work closely with researchers and cross-functional stakeholders to translate research ideas and technical requirements into scalable engineering solutions.
-
Specialist Expertise: Deep experience in at least one of the following areas:
Large-Scale Reinforcement Learning Training
-
Multi-node GPU training using PyTorch and technologies such as Megatron-Core, FSDP/DTensor, or DeepSpeed.
-
Understanding of tensor, pipeline, context, and expert parallelism strategies, including their trade-offs for Mixture-of-Experts and long-context models.
-
Familiarity with LLM inference engines such as vLLM or SGLang.
-
GPU profiling experience involving memory, NCCL, and CUDA graphs.
Evaluation Engineering
-
Experience building, scaling, and optimizing evaluation suites for large language models.
-
Ability to demonstrate the measurable impact of evaluation engineering work.
-
Lead-Level Experience: For Lead-level candidates, at least 3 years of professional experience in the relevant technical area, together with a track record of leading complex technical projects end to end and mentoring engineers.
-
Additional Preferred Experience:
-
Building or extending RL post-training frameworks such as NeMo-RL, verl, OpenRLHF, or TRL, particularly asynchronous or non-colocated architectures.
-
Familiarity with evaluation frameworks such as Inspect, Harbor, or lm-evaluation-harness.
-
Experience with distributed runtimes and orchestration tools such as Ray, Kubernetes, or Slurm across AWS, Azure, or Google Cloud.
-
Significant contributions to open-source projects such as vLLM, Megatron, or PyTorch.
-
Familiarity with AI-assisted software development and using coding agents at scale.
Candidates are encouraged to apply even if they do not meet every preferred qualification, as relevant experience and potential alignment will also be considered.
Benefits:
-
Base Salary: For eligible U.S. locations, the annual base salary range is $118,400–$219,800 USD. Actual compensation depends on location, skills, knowledge, experience, and internal equity.
-
Annual Bonus: The role may be eligible for an annual bonus based on a combination of company and individual performance.
-
Health Coverage: Medical, dental, and vision insurance, as well as disability and life insurance programs.
-
Retirement Savings: 401(k) plan with an employer match.
-
Time Off: Flexible vacation, paid sick and safe time off, paid holidays, and two company-wide Mental Health Days.
-
Family Support: Parental leave and adoption and surrogacy assistance.
-
Additional Insurance Options: Optional hospital, accident, sickness, life, and AD&D insurance plans, paid fully by the employee.
-
Financial and Wellness Resources: Flexible Spending Accounts, Health Savings Accounts, fitness reimbursement, Employee Assistance Program, and access to Headspace.
-
Professional Development: Tuition reimbursement, career development programs, continuous learning opportunities, and skills-based growth resources.
-
Additional Programs: Employee incentive programs, Employee Stock Purchase Plan, commuter benefits, 529 Plan access, and optional group legal and identity theft protection.
-
Flexibility and Work-Life Balance: Remote work arrangements and the option to work from anywhere for up to eight weeks per year, subject to applicable policies.
-
Community Impact: Two paid volunteer days annually, with opportunities to participate in pro-bono projects and environmental, social, and governance initiatives.
-
Work Location: Remote positions are available in the United States, including Eagan, Minnesota; Frisco, Texas; New York, New York; and Ann Arbor, Michigan. The original posting also lists Toronto, Ontario, Canada.
-
Recruitment Timeline: The posting is scheduled to close on March 31, 2027. It may also be used to support future vacancies.
Skills
As published by lever
Resume/CV, Full name, Email, Phone, Current location, Current company
Sign in to read and join the discussion.