GPU Infrastructure Support Engineer

GPU Infrastructure Support Engineer About Civo: Civo is reimagining the cloud industry by providing developers and businesses with cutting-edge, developer-friendly cloud solutions. With a focus on simplicity, performance, and reliability, Civo enables its clients to unleash the full potential of their projects and scale with ease. As a company committed to excellence and innovation, Civo values diversity, collaboration, and continuous improvement. About the Role: GPU Infrastructure Support Engineer, you will serve as the frontline operational defense for Civo’s global GPU and AI infrastructure fleet. You will handle day-to-day customer tickets, platform alerts, hardware fault triage, and initial incident response across GPU nodes, high-speed networking fabrics, and Linux systems. This is a hands-on, high-impact role requiring strong technical curiosity and structured troubleshooting discipline. You will own incidents end-to-end, conduct initial hardware diagnostics, coordinate smart-hands activities in our data centers, and ensure clean, evidence-rich handovers when escalating complex issues to Senior GPU Infrastructure Engineers. Responsibilities: 24/7 Operations & Ticket Triage - Incident Triage: Monitor platform dashboards and queue systems to acknowledge, prioritize, and triage customer-reported issues within strict SLA targets. - Customer Communication: Provide clear, professional, and technical status updates to enterprise clients via ticketing channels and status pages. - Shift Handovers: Maintain accurate event logs and execute clean, detailed shift handovers to ensure continuous operational coverage. GPU Node & Hardware Triage - First-Line Diagnostics: Execute runbooks to inspect node health using nvidia-smi, NVIDIA Data Center GPU Manager (DCGM), and system event logs (dmesg, journalctl). - Out-of-Band Management: Access and manage bare-metal host BMCs via IPMI or Redfish to perform power cycles, remote console inspections, and system event log extractions. - RMA Preparation: Gather diagnostic logs and serial numbers to build complete, evidence-rich vendor RMA submission packages for failed GPUs, DIMMs, or storage drives. Data Center & Field Support Coordination - Smart-Hands Coordination: Act as the primary technical contact for local data center facility technicians, directing physical component reseats, optic replacements, and cable testing. - Asset Tracking: Maintain absolute data integrity across Civo’s DCIM and IPAM systems (e.g., NetBox) during hardware swaps or maintenance. Platform & Environment Troubleshooting - User Support: Assist customer teams with basic SSH access, GPU device passthrough checks, Slurm job queue status, and container runtime issues. - Runbook Execution & Improvement: Follow operational runbooks for routine maintenance and propose incremental updates based on real-world incident trends. Key Results/Objectives: - SLA Adherence: Maintain >98% compliance on initial response and update SLAs across all priority tiers. - First-Contact Resolution: Resolve >= 60% of tier-1 platform and access tickets without requiring L3 escalation. - RMA Processing Speed: Package and submit verified RMA evidence to hardware vendors within 4 hours of fault confirmation. Requirements: Experience & Core Qualifications - 2–4+ years of technical experience in an IT infrastructure support, service desk, or data center operations role within a structured, customer-facing environment. - Demonstrated hands-on experience troubleshooting server hardware and Linux operating systems. Technical Expertise - Linux Administration: Strong CLI proficiency with systemd, file permissions, storage mounts, and network diagnostic utilities (ip, ping, traceroute, netstat). - GPU & Hardware Diagnostics: Working knowledge of NVIDIA GPU server architectures, including command-line tools like nvidia-smi and basic DCGM health checks. - Hardware & Out-of-Band: Familiarity with server architecture (PCIe, NVMe, memory topology) and Out-of-Band management (iDRAC, iLO, BMC via IPMI/Redfish). - Ticketing & ITIL: Working understanding of incident management workflows, prioritization, and ticket lifecycle tracking. Soft Skills - Exceptional written communication skills with the ability to summarize complex technical logs into clear customer updates and internal escalation notes. - Strong organizational discipline and a methodical approach to step-by-step technical isolation. Nice to Have: - Experience with Slurm workload manager or Kubernetes (kubectl basics). - Basic awareness of high-performance networking concepts (InfiniBand, RoCEv2, RDMA). - Linux certifications (RHCSA, CompTIA Linux+) or Kubernetes foundations (CKA/KCNA). Why Join Civo? - Competitive compensation and benefits package. - 4-day work week company (unless attending an event). - Uncapped holiday. - Remote work environment with flexibility and autonomy. - Collaborative and inclusive culture that values diversity and creativity. - Opportunity to work with a dynamic and innovative team in the fast-growing cloud industry.