Senior GPU Infrastructure Engineer
Posted
Description
Senior GPU Platform Reliability Engineer (AI & GPU Infrastructure)
Join a high-growth GPU Neo-cloud challenging hyperscaler cost and complexity. As the technical authority for the reliability, performance and automated orchestration of its AI and GPU platform, you’ll keep large-scale training and inference clusters running at maximum uptime and build the tooling that removes hardware entropy at scale.
Responsibilities
- Act as the top internal escalation point for hardware, PCIe, GPU memory and driver faults across the GPU fleet, owning deep root-cause analysis
- Build automated pre-job and post-job health checks using telemetry and benchmarking tools
- Own automated rollout pipelines for firmware, BIOS and driver stacks on live clusters with zero unannounced customer impact
- Diagnose bottlenecks and link degradation across InfiniBand and RoCEv2 fabrics, and optimise storage data paths
- Maintain Kubernetes and Slurm GPU tooling, with automated node-cordoning and repair workflows
- Drive Infrastructure-as-Code (Terraform, Ansible, Python) for fleet expansion, and lead major incident resolution and blameless post-mortems
- Work directly with NVIDIA and hardware OEM engineering teams on vendor escalations
Must have
- 5+ years in SRE, Systems or Infrastructure Engineering on production-scale Linux
- 3+ years supporting data centre GPUs in distributed environments
- Strong troubleshooting skills across GPU hardware, drivers and interconnects
- Advanced Kubernetes (CKA level) and Slurm
- InfiniBand or RoCEv2 networking knowledge and NCCL tuning
- Strong Python, Go or Bash, plus Terraform and Ansible
- Prometheus and Grafana observability experience
- Calm leadership in high-pressure incidents and excellent technical writing
Nice to have
- Hands-on experience with the NVIDIA stack (CUDA, DCGM, NVLink/NVSwitch, XID troubleshooting)
- NVIDIA certifications (NCP-AII, NCP-IB)
- CKA or CKS
- Liquid-cooled rack experience (Direct-to-Chip, CDU)
Benefits
- 4-day working week
- Uncapped holiday
- Remote-first with real autonomy
- Competitive compensation and benefits
- Blameless, engineering-led culture with direct access to NVIDIA and OEM engineers
Ready to keep some of the world’s most in-demand GPU infrastructure running at its best? Apply now to join the team.
Skills
Sign in to read and join the discussion.