Remote ML Evaluation & Experiment Design Engineer
Posted Updated
Description
Anyone AI Inc. is seeking experienced Machine Learning Engineers to review and evaluate ML challenges used in AI model training and assessment. You will analyze experiments, datasets, metrics, and pipelines to ensure challenges are technically sound, reproducible, and demand genuine ML reasoning, not brute-force tuning.
This is a remote, part-time, project-based engagement. You’ll provide clear feedback on design choices, detect data leakage, and help improve evaluation protocols.
Review ML challenges and determine if they are well designed and solvable. Evaluate whether datasets contain meaningful, learnable signals. Identify unintended shortcuts or artifacts in synthetic datasets. Determine whether tasks require genuine diagnosis of the underlying ML problem. Review evaluation metrics and improvement thresholds. Detect metric gaming, data leakage, and evaluation flaws. Verify reproducibility across data → model → evaluation pipeline. Assess whether challenge difficulty is appropriately calibrated. Provide clear recommendations for improving, recalibrating, or excluding problematic tasks. Hands-on ML experience with designing and evaluating experiments. Ability to assess whether signals are learnable and robust. Strong sense for identifying data leakage and distribution shift. Proficiency in selecting appropriate evaluation metrics. Clear written feedback on technical problems.Skills
Sign in to read and join the discussion.