Senior NVIDIA Network Engineer – InfiniBand / Ethernet
Posted
Description
Hiring: Senior NVIDIA Network Engineer – InfiniBand / Ethernet
Location: Chennai, India
Experience: 7–15 Years
Work Mode: Onsite/ Hybrid
Schedule: On-call support for P1 network incidents
About the Role
We are seeking an experienced NVIDIA Network Engineer with strong expertise in InfiniBand (HDR/NDR), RoCE v2, and high-performance data center networking.
You will design, deploy, optimize, and support ultra-low-latency network fabrics powering large-scale NVIDIA GPU clusters (approximately 10K GPUs) for AI/ML training and inference workloads.
Must-Have Technical Skills
- InfiniBand Networking: HDR/NDR, IB routing, credit management, timers, fabric troubleshooting, and congestion control.
- High-Performance Ethernet: RoCE v2, PFC, ECN, MTU, buffer tuning, QoS, and GPUDirect RDMA.
- Network Architecture: Leaf-spine, ECMP, east-west InfiniBand fabrics, north-south Ethernet connectivity, BGP, and EVPN.
- GPU Cluster Networking: NVIDIA GPU infrastructure, NCCL collectives, all-reduce optimization, UCX, SHARP offloads, and distributed AI training communications.
- NVIDIA Networking Platforms: NVIDIA Spectrum/Quantum switches, UFM, NetQ (or equivalent), Cumulus, Onyx, and SONiC.
- Network Automation: Ansible, Terraform, configuration automation, and change control.
- Performance Monitoring: iperf, NCCL-tests, sFlow, telemetry, throughput/latency monitoring, and congestion analysis.
- Network Operations: Switch OS/firmware lifecycle, optics and cable qualification, inventory, spares, and day-2 fabric health.
- Incident Management: P0/P1 incident response, path failures, fabric partitions, link flaps, slow-drain, credit starvation, RCA, and post-incident improvements.
- Security: BGP/EVPN multi-tenancy, ACLs, role-based fabric manager access, configuration signing, and change control.
Key Responsibilities
- Build and operate InfiniBand HDR/NDR and Ethernet network fabrics supporting large-scale GPU clusters.
- Configure and optimize RoCE v2, PFC/ECN, MTU, buffer profiles, IB routing, and SHARP/UCX offloads.
- Validate NCCL collective performance and troubleshoot all-reduce bottlenecks across GPU pods.
- Implement QoS classes for training, inference, and control-plane traffic.
- Monitor fabric congestion, slow-drain, link failures, and network stability.
- Automate network configurations using Ansible and Terraform.
- Lead P0/P1 incidents involving InfiniBand, RoCE, Ethernet, NCCL, and GPUDirect RDMA.
- Develop operational runbooks, improve network resilience, and implement graceful degradation.
- Track fabric throughput/latency SLOs, congestion events, NCCL throughput variance, and change success rates.
Qualifications
- BE/B-Tech or equivalent in Computer Science or Electronics & Communication.
- 7–15 years of data center networking experience.
- Deep hands-on expertise in InfiniBand HDR/NDR and RoCE tuning at scale.
- Strong understanding of GPU cluster communication patterns and distributed AI/ML workloads.
Preferred Certifications
- NVIDIA Networking Certification
- CCNP Data Center / CCIE Data Center
- JNCIP-DC
- Arista ACE
Skills
Sign in to read and join the discussion.