Operation Engineer (AI/ HPC Platform)
Posted
Description
Operations Engineer (AI/HPC)
Role Overview
Provides first- and second-line operational support for a large-scale AI and High-Performance Computing (HPC) platform. You will troubleshoot alerts, restore services, and maintain daily operations across Linux servers, Kubernetes, GPU clusters, networking, and storage, escalating complex faults to Senior Engineers.
Core Responsibilities
- Incident & Alert Response: Monitor system alerts, diagnose service failures, maintain clear ticket logs, and escalate major issues with detailed evidence.
- Linux & Kubernetes Operations: Troubleshoot unhealthy nodes, failed pods, resource bottlenecks, DNS/networking issues, and shared storage permissions.
- GPU & Hardware Maintenance: Monitor GPU health, isolate faulty nodes, run approved patching/reprovisioning runbooks, and coordinate hardware repairs with vendors.
- Compliance & Documentation: Execute approved user access/configuration changes under strict change control (GxP) and keep operational runbooks and automation scripts up to date.
Must-Have Skills
- Linux & Containers: Hands-on production Linux administration (CLI diagnostics, logs, networking, storage mounts) and working knowledge of Kubernetes and containers.
- Networking & Storage: Solid grasp of TCP/IP, DNS, routing, and shared filesystems (like NFS).
- Troubleshooting & Scripting: Structured problem-solving using monitoring/ticketing tools, basic shell scripting, and familiarity with Git/version control.
Nice-to-Have Skills
- Experience with NVIDIA GPUs (CUDA, DCGM, GPU Operator, Run:ai) and AI/HPC workloads.
- Familiarity with Prometheus/Grafana, Ansible, Python, Helm, GitOps, scale-out storage (e.g., VAST), and server BMC tools (e.g., HPE iLO).
Skills
Sign in to read and join the discussion.