Lead AI/ML Engineer
Description
Lead AI/ML Engineer
IG Labs · AI Platform Engineering
About the Role:
You are a senior individual contributor on the AI/ML engineering team, responsible for how capable and reliable our agents actually are. That means owning the systems that make an agent smart — long-term memory, advanced retrieval, and the model adaptation (fine-tuning and reinforcement loops) behind them — while holding the evaluation bar that decides what ships. As a Staff engineer you choose the right approach for each problem, own the hardest ones end-to-end, and raise the bar for the engineers around you. The work is empirical and equal parts R&D and software engineering.
Key Responsibilities:
● Set technical direction for agent intelligence — for each agent-quality problem, decide whether the right lever is model adaptation, long-term memory, retrieval, or context engineering, and defend the call on evals and cost. In this landscape the naive move (just fine-tune) is usually wrong; the sophisticated one is knowing when.
● Own the long-term memory systems — episodic, semantic, and procedural memory; consolidation, forgetting, and retrieval of memory; and working-memory / context management so agents stay coherent across long, multi-session tasks.
● Advance the retrieval stack — agentic RAG, hybrid retrieval with reranking, late-interaction and contextual retrieval, query decomposition, and knowledge-graph-backed retrieval — every choice pushed on real evaluations.
● Own model adaptation where it pays — fine-tuning (SFT / LoRA / QLoRA), preference and reinforcement-learning loops (reward modeling, RLHF / RLAIF / RLVR), and distillation of smaller specialized models — and the judgment to know when not to train.
● Engineer context end-to-end — govern what enters the model's window at each step: retrieved knowledge, memory, tool results, and compaction. This is the discipline that most determines agent quality.
● Own the evaluation bar — trajectory-level and outcome evals across memory, retrieval, and models; regression suites that gate what ships; offline / online alignment; and the human-in-the loop labeling behind ground truth.
● Make it production-grade — instrument agents for observability (tracing, behavior and cost/latency drift), drive reliability, and own the quality / cost / latency tradeoff as agents make many model calls per task.
● Push the R&D frontier — evaluate and de-risk new methods (memory architectures, retrieval techniques, RL algorithms, scalable oversight) and turn the ones that pay off into production capabilities of the Agentic SDK.
● Own model- and agent-behavior safety — red-team for prompt injection, jailbreaks, memory poisoning, and untrusted-content handling; enforce per-tenant isolation across the memory, retrieval, and model layers; and hold the data-rights boundaries that govern training.
● Multiply the team — set the patterns others build on, review designs and eval methodology, mentor Senior and AI/ML engineers, and be the person others bring hard model-quality problems to.
Required Qualifications:
● 8+ years of software / ML engineering, including several years building LLM-powered agentic systems in production with end-to-end ownership from design through operation.
● Deep, hands-on strength in at least two of: model adaptation (fine-tuning with LoRA / QLoRA, preference / RL methods, reward modeling); long-term memory and context management for agents; advanced retrieval (hybrid + reranking, agentic / contextual RAG, query understanding); and evaluation methodology for models and agents — with working competence across the rest.
● Judgment about which lever fits — can take an agent-quality problem and decide, with evidence, whether to fine-tune, build memory or retrieval, or re-engineer context, and defend it on evals and cost.
● Rigorous evaluation practice — has designed eval methodology, including trajectory- or outcome level evals for agents, that gates what ships and caught regressions informal testing missed.
● Strong Python and systems engineering — builds the pipelines and services, not just the models; comfortable with async code, data pipelines, and integrating into a production SDK.
● Fluent in the modern stack — PyTorch and the HuggingFace ecosystem on the model side; vector / hybrid retrieval and reranking on the systems side; and the observability tooling to trace and debug agent behavior.
● Can drive ambiguous, cross-cutting model-quality problems to production without the scope being fully defined upfront.
● A track record as a force multiplier — technical leadership, design and code review, and mentoring other engineers.
Preferred Qualifications:
● Production long-term memory systems — episodic / semantic / procedural memory, consolidation and forgetting, and memory retrieval at scale.
● Reinforcement-learning depth — has run policy optimization (PPO, GRPO, DPO-family, RLHF / RLAIF, or RLVR) and reward modeling end-to-end, and understands reward hacking and training instability.
● Advanced retrieval depth — late-interaction (ColBERT), GraphRAG or knowledge-graph retrieval, contextual retrieval, or RAG systems tuned on real evals.
● Has built fine-tuning pipelines used by other engineers — reproducible training with data versioning, config management, checkpointing, and automated eval gates.
● Multi-domain adaptation — per-domain adapters, model merging (SLERP / TIES / DARE), routing, and small specialized models.
● Agent security — defending against prompt injection, memory poisoning, and untrusted-content handling in tool- and memory-enabled agents.
● Cost / latency optimization for multi-call agent workloads — caching, model routing, and high throughput serving (e.g., vLLM with adapter routing).
● Synthetic data and scalable oversight — self-instruct / Evol-Instruct, distillation, LLM-as-judge, constitutional AI / RLAIF.
● Open-source contributions to relevant projects — TRL, HuggingFace, vLLM, LangGraph, retrieval or memory frameworks, or comparable.
Skills
Sign in to read and join the discussion.