Lead AI Engineer
Summary
Build and maintain end-to-end training data pipelines for Large Language Models (LLMs), covering data cleaning, deduplication, quality filtering, and versioning. Core technologies: Python, pandas, LLM SFT/RLHF data pipelines.
Job Description
- Build and maintain end-to-end training data pipelines for Large Language Models (LLMs), covering data cleaning, deduplication, format conversion, quality filtering, data mixture/formulation, and version management.
- Design and dynamically adjust the data mix ratio between SFT training data and general-purpose Replay data.
- Manage the Prompt pool for distillation training, including prompt collection, deduplication, and balanced sampling across different domains.
- Establish a comprehensive data quality assurance framework, implementing a two-level automated and manual sampling-based quality control process covering format validation, semantic deduplication, and persona consistency checks.
- Collaborate with Agent Algorithm Engineers to convert ReAct multi-turn reasoning trajectories and Function Calling workflows into training-ready data formats.
- Manage training data versioning and traceability, ensuring that data snapshots and change records for each training iteration are fully documented and traceable.
Job Requirements
- Bachelor’s degree or above in Computer Science, Artificial Intelligence, Data Science, or a related field.
- Proficient in Python and familiar with data processing frameworks such as pandas, with hands-on experience processing large-scale text datasets.
- Experience building LLM SFT/RLHF data pipelines, with a solid understanding of best practices in data deduplication, quality filtering, and data mixture/formulation.
- Familiarity with ChatML/Chat formats, Function Calling data formats, and multi-turn conversational data structures.
- Experience in data quality measurement, monitoring, and data version management.