AI Systems Engineer
Description
The AI Systems Support Engineer is responsible for the reliability, availability, and resilience of DAMAC AI's GPU cluster infrastructure — ensuring that B300 and GB300 Blackwell-based DGX SuperPOD and HGX systems achieve and exceed defined SLA and uptime commitments. This role combines traditional site reliability engineering with deep expertise in GPU infrastructure failure modes, disaster recovery, business continuity, and the operational resilience of mission-critical AI compute environments.
Reliability Engineering & SLA Management
Define, measure, and report Service Level Objectives (SLOs) and Service Level Indicators (SLIs) for DAMAC AI's GPU cluster infrastructure — compute availability, GPU health, storage throughput, fabric latency, and workload completion rates
Maintain infrastructure uptime ≥99.9% across DGX SuperPOD and HGX GPU cluster environments through proactive fault management and preventive maintenance
Own the error budget framework — tracking SLO burn rate, escalating when error budget is at risk, and freezing changes during critical burn periods
Implement reliability improvement initiatives using data from post-incident reviews, SLO trends, and GPU telemetry
Produce weekly and monthly infrastructure reliability reports for VP Technology and executive leadership
Disaster Recovery & Business Continuity
Design and maintain Disaster Recovery (DR) and Business Continuity Plans (BCP) for all GPU cluster environments — covering compute, storage, networking, and power/cooling failure scenarios
Define Recovery Time Objectives (RTOs) and Recovery Point Objectives (RPOs) for all critical AI infrastructure components and customer workloads
Architect and test multi-site resilience strategies for DAMAC AI's global data center portfolio — including active-active, active-passive, and warm standby configurations
Conduct quarterly DR simulation exercises — full failover tests covering GPU cluster loss, storage failure, fabric partition, and power outage scenarios
Maintain runbooks for all major DR scenarios — GPU node failure, InfiniBand fabric partition, storage cluster loss, and full site outage
Incident Management & Root Cause Analysis
Lead P1 and P2 incident response for GPU infrastructure events — detection, triage, escalation, containment, recovery, and post-incident review
Chaos Engineering & Resilience Testing
Design and execute chaos engineering experiments targeting GPU infrastructure failure modes — GPU node loss, NVLink failure, InfiniBand switch failure, storage path loss, and power PDU failure
Use chaos engineering frameworks (Chaos Monkey, Litmus, custom scripts) to validate the resilience of GPU cluster deployments under real failure conditions
Validate that NVIDIA DGX SuperPOD redundancy features — dual InfiniBand fabrics, RAID-protected NVMe, redundant power — perform as specified under simulated failure
Test and validate AI workload checkpoint/restart capabilities to ensure training job resilience against node failures
Document chaos test results and drive architecture improvements based on identified weaknesses Capacity Planning & Performance Management
Own GPU infrastructure capacity planning — forecasting compute, storage, and network capacity requirements against customer workload growth
Monitor and report GPU cluster resource utilisation trends — compute, memory, storage, and fabric — to inform procurement and expansion planning
Identify performance bottlenecks across GPU compute, NVLink/InfiniBand fabric, and storage tiers — partnering with AI Systems and Network Engineers to resolve
Skills
Sign in to read and join the discussion.