Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

DAMAC Digital

NewBe an early applicant

AI Systems Engineer

Posted

Description

The AI Systems Support Engineer is responsible for the reliability, availability, and resilience of DAMAC AI's GPU cluster infrastructure — ensuring that B300 and GB300 Blackwell-based DGX SuperPOD and HGX systems achieve and exceed defined SLA and uptime commitments. This role combines traditional site reliability engineering with deep expertise in GPU infrastructure failure modes, disaster recovery, business continuity, and the operational resilience of mission-critical AI compute environments.


Reliability Engineering & SLA Management

Define, measure, and report Service Level Objectives (SLOs) and Service Level Indicators (SLIs) for DAMAC AI's GPU cluster infrastructure — compute availability, GPU health, storage throughput, fabric latency, and workload completion rates

Maintain infrastructure uptime ≥99.9% across DGX SuperPOD and HGX GPU cluster environments through proactive fault management and preventive maintenance

Own the error budget framework — tracking SLO burn rate, escalating when error budget is at risk, and freezing changes during critical burn periods

Implement reliability improvement initiatives using data from post-incident reviews, SLO trends, and GPU telemetry

Produce weekly and monthly infrastructure reliability reports for VP Technology and executive leadership


Disaster Recovery & Business Continuity

Design and maintain Disaster Recovery (DR) and Business Continuity Plans (BCP) for all GPU cluster environments — covering compute, storage, networking, and power/cooling failure scenarios

Define Recovery Time Objectives (RTOs) and Recovery Point Objectives (RPOs) for all critical AI infrastructure components and customer workloads

Architect and test multi-site resilience strategies for DAMAC AI's global data center portfolio — including active-active, active-passive, and warm standby configurations

Conduct quarterly DR simulation exercises — full failover tests covering GPU cluster loss, storage failure, fabric partition, and power outage scenarios

Maintain runbooks for all major DR scenarios — GPU node failure, InfiniBand fabric partition, storage cluster loss, and full site outage

Incident Management & Root Cause Analysis

Lead P1 and P2 incident response for GPU infrastructure events — detection, triage, escalation, containment, recovery, and post-incident review


Chaos Engineering & Resilience Testing

Design and execute chaos engineering experiments targeting GPU infrastructure failure modes — GPU node loss, NVLink failure, InfiniBand switch failure, storage path loss, and power PDU failure

Use chaos engineering frameworks (Chaos Monkey, Litmus, custom scripts) to validate the resilience of GPU cluster deployments under real failure conditions

Validate that NVIDIA DGX SuperPOD redundancy features — dual InfiniBand fabrics, RAID-protected NVMe, redundant power — perform as specified under simulated failure

Test and validate AI workload checkpoint/restart capabilities to ensure training job resilience against node failures

Document chaos test results and drive architecture improvements based on identified weaknesses Capacity Planning & Performance Management

Own GPU infrastructure capacity planning — forecasting compute, storage, and network capacity requirements against customer workload growth

Monitor and report GPU cluster resource utilisation trends — compute, memory, storage, and fabric — to inform procurement and expansion planning

Identify performance bottlenecks across GPU compute, NVLink/InfiniBand fabric, and storage tiers — partnering with AI Systems and Network Engineers to resolve


Skills

Apply

See also

Software Engineering jobs by country — openings, pay and top skills →