Staff GPU Inference SDET
Posted Updated 9
views
Summary
Staff-level SDET building automated test frameworks, regression gates, and release-qualification pipelines for GPU inference and distributed LLM serving systems. Day to day involves benchmarking, fault injection, numerical-correctness validation, and CI/CD using Python, Kubernetes/Slurm/Ray, and InfiniBand/NCCL.
You will build automated test frameworks, regression gates, and release qualification pipelines for GPU inference systems. You will validate distributed serving workloads, performance models, numerical correctness, resilience, and recovery; integrate observability and CI/CD; and investigate software and hardware failures.
Responsibilities
- Design automated test frameworks, regression gates, and release qualification pipelines
- Benchmark and stress-test distributed LLM serving frameworks
- Build workload replay and benchmarking tools
- Validate model accuracy, precision stability, determinism, and output correctness
- Engineer fault-injection and recovery tests for GPU clusters
- Integrate test pipelines with telemetry and continuous monitoring
Requirements
- Eight or more years of software engineering experience as an SDET, infrastructure quality lead, or systems test engineer
- Experience bringing up, provisioning, and validating multi-node GPU clusters
- Knowledge of LLM serving engines, prefill and decode disaggregation, KV-cache management, and dynamic batching
- Expert Python programming and test automation framework experience
- Kubernetes, Slurm, or Ray
- InfiniBand, RoCE, or NCCL
- Root-cause analysis, stress testing, and node failure simulation