Senior ML Inference Systems Engineer | Remote & Equity
Posted Updated
Description
Cast AI is seeking a senior ML inference infrastructure engineer to optimize throughput, latency, and cost across our GPU fleet. You will tune loaders, schedulers, and KV caches, ensuring scalable, multi-node inference while maintaining model quality.
You will own the technical direction, work with vLLM/SGLang/TensorRT-LLM stacks, and collaborate across a global, remote-first team. A strong production background is essential.
Push throughput: continuous batching, prefill, drafting models, CUDA-graph capture, scheduler tuning across the engine fleet. Cut latency: profile to find bottlenecks and fix compute, memory, scheduling, or storage issues. Get more out of the KV cache: hierarchical/paged attention, prefix caching, eviction policies, quantized KV. Kill cold starts: weight loaders, JIT and compile caches, shared-filesystem paths. Quantize without regressing quality: INT8/INT4, FP8/FP4 across weights/activations/KV. Scale across nodes: distributed inference, network-aware placement, checkpointing strategies. You've built real ML systems for 5+ years, with depth in inference or training infrastructure. Your Python is production-grade: services, not scripts. You've run vLLM, SGLang, or TensorRT-LLM for real and understand performance. You've measured quantization regressions, not just compression ratios. Multi-GPU and multi-node setups are familiar: tensor/expert parallelism and communication. You instrument before you optimize and can identify real wins from benchmarks. Equity options Remote-first Learning budget Annual hackathon Team events Equipment budget Extra days offSkills
Sign in to read and join the discussion.