Lead Systems Software Engineer - GPU & AI Infrastructure

Role Overview: We are looking for a Lead Systems Software Engineer (GPU Infrastructure) to architect, deploy, and scale the software platform powering our newly established GPU data center cluster. You will bridge the gap between bare-metal accelerated hardware and the software stack required to run massive AI training, inference, or high-performance computing (HPC) workloads.

Your primary responsibility will be creating a highly resilient, automated, and observable software layer that maximizes GPU utilization, eliminates networking bottlenecks, and offers seamless scheduling capabilities to internal or external development teams.

Core Responsibilities

• Cluster Orchestration & Scheduling: Design, implement, and maintain multi-node GPU clustering software using orchestration frameworks like Kubernetes (with advanced scheduling plugins like Volcano, Kueue) or Slurm to handle heavy batch processing workloads.
• Hardware-to-Software Optimization: Write, configure, and optimize low-level drivers, runtimes, and libraries (e.g., NVIDIA CUDA, NCCL, Triton Inference Server) to ensure deep learning frameworks can communicate across the cluster with zero latency.
• Performance Bottleneck Analysis: Identify and eliminate systemic performance drops across host CPUs, PCIe buses, high-speed interconnects (InfiniBand/RoCE), and high-performance storage systems (e.g., Lustre, GPFS, Weka).
• Infrastructure Automation (IaC): Build robust automation scripts and configuration management tools using Ansible, Terraform, and Helm to provision, patch, and deploy bare-metal server fleets dynamically.
• Telemetry & Observability: Develop and maintain specialized telemetry systems (using Prometheus, Grafana, and DCIM integrations) to monitor GPU temperatures, power consumption, memory errors (XIDs), and structural utilization metrics.
• Collaboration: Act as the technical translator between the data center facility engineering team (managing power and liquid cooling) and the product engineering teams (running the AI models).

Required Technical Skills & Qualifications

• Experience: 5+ years in Systems Engineering, Infrastructure Engineering, or low-level DevOps, with at least 2 years explicitly handling production-scale GPU or HPC workloads.
• Programming: Strong proficiency in Python, Go, or C++ for writing system tooling, automation, or custom infrastructure components.
• Linux & Networking: Master-level knowledge of production Linux (Ubuntu/RedHat), networking internals (TCP/IP, routing, firewalls), and advanced cluster networking like InfiniBand or RoCE (RDMA over Converged Ethernet).
• Containerization Stack: Deep experience managing production Kubernetes environments, Container Runtime Interfaces (CRI), and NVIDIA Container Toolkit integrations.
• Hardware Literacy: Familiarity with modern data center GPU architecture (e.g., NVIDIA H100/B200 platforms), high-bandwidth memory (HBM), and interconnect systems like NVLink.

Preferred / "Nice to Have" Qualifications

• Experience implementing liquid-cooling software alerts or digital twin facility
• telemetry.
• Contributions to open-source distributed training frameworks or cloud-native batch scheduling projects.
• Certified Kubernetes Administrator (CKA) or formal NVIDIA/HPC systems architecture certifications