NVIDIA GPU Infrastructure Engineer / HPC Engineer
Posted
Description
Hiring: NVIDIA GPU Infrastructure Engineer / HPC Engineer
Location: Chennai, India
Experience: 7–12 Years
Work Mode: Onsite / Hybrid
About the Role
We are looking for an experienced NVIDIA GPU Infrastructure Engineer / HPC Engineer to build, manage, and optimize large-scale AI/HPC GPU clusters supporting high-performance LLM training and inference workloads.
The ideal candidate will have strong hands-on expertise in NVIDIA H100/B-series GPUs, CUDA, Slurm, Kubernetes, NCCL, and GPU performance optimization, with experience supporting enterprise-scale GPU infrastructure.
Must-Have Technical Skills
- GPU Infrastructure: NVIDIA H100/B-series GPUs, large-scale GPU clusters (up to 10K GPUs), multi-pod architecture, GPU/CPU/Memory/NIC capacity planning, and heterogeneous GPU SKU bin-packing.
- GPU Virtualization: NVIDIA MIG, vGPU profiles, multi-tenant GPU partitioning, quota management, and resource allocation.
- GPU Computing: CUDA, cuDNN, TensorRT, NCCL, UCX, GPU clocks/persistence, and CUDA runtime tuning.
- Cluster Orchestration: Slurm, Kubernetes (K8s), GPU Device Plugins, DaemonSets, Node Feature Discovery (NFD), scheduling, accounting, and quotas.
- GPU Monitoring: NVIDIA DCGM, NVML, nvidia-smi, Prometheus, Grafana, ELK, and Splunk.
- Automation & Deployment: Helm, ArgoCD, golden images, node remediation (cordon/drain/reimage), and auto-healing workflows.
- High-Performance Networking: InfiniBand (IB), Ethernet, RDMA, TOR/leaf connectivity, NUMA/locality optimization, and GPUDirect Storage.
- Performance Benchmarking: HPL, HPL-AI, MLPerf-like benchmarking, performance baselines, regression analysis, and SLO monitoring.
- Infrastructure Operations: Rack/power/cooling layouts, firmware and driver lifecycle management, change management, and zero/low-downtime maintenance.
- Security: MIG isolation, cgroups/device cgroups, mpsd, secure GPU drivers/containers, SBOM, and image scanning.
Key Responsibilities
- Deploy, configure, and operate enterprise-scale NVIDIA GPU clusters supporting AI/ML training and inference.
- Integrate Slurm/Kubernetes scheduling with GPU resource management and multi-tenant workloads.
- Optimize CUDA, NCCL, UCX, RDMA, GPU performance, and distributed training efficiency.
- Manage GPU firmware, drivers, monitoring, cluster health, and capacity planning.
- Lead P0/P1 production incidents involving GPU, CUDA, networking, NCCL, RDMA, and scheduling failures.
- Coordinate with Networking, GPU Operations, Platform Engineering, Storage, and Application teams to restore services within SLA targets.
- Perform root cause analysis (RCA), implement corrective actions, and improve infrastructure reliability.
- Maintain runbooks, performance baselines, and application tuning documentation.
- Track GPU utilization, queue wait-time SLOs, failed-job rates, performance baseline adherence, and incident MTTD/MTTR.
Qualifications
- BE/B-Tech or equivalent in Computer Science or Electronics & Communication.
- 7–12 years of hands-on experience building and operating HPC/AI GPU clusters at scale.
- Deep expertise in CUDA, MIG, and vGPU.
- Proven experience with NVIDIA H100/B-series GPUs, Slurm and/or Kubernetes device scheduling, and NCCL/UCX performance tuning.
Preferred Certifications
- NVIDIA Certified Professional – AI Infrastructure
- RHCE / LPIC (Linux).
Skills
Sign in to read and join the discussion.