Senior Infrastructure Engineer
Posted Updated
Description
- We are seeking a Software Engineer to join Crusoe’s Hardware Systems Engineering team and build the tools, automation, and data platforms that let our hardware engineers debug, validate, and support high-performance compute systems at scale
- In this role, you will own the software that supports the full hardware lifecycle, from prototype bring-up through large-scale production, accelerating issue resolution and improving reliability across Crusoe Cloud’s GPU- and CPU-based infrastructure
- You will work closely with hardware engineers to understand their debugging and validation workflows, then turn them into automatable workflows
- You will also work with cross-functional teams to integrate tools with the production environment
- A particular focus will be provisioning, hardware health monitoring, test framework, and data collection
- Your work will directly improve Crusoe’s ability to deploy and operate sustainable, AI-first compute systems with world-class performance and reliability
- Design, build, and maintain software tools and frameworks that support the hardware development and sustaining lifecycle, including bring-up, validation, deployment, and ongoing production support
- Develop automated test, diagnostics, and health-monitoring frameworks that run across fleets of servers, enabling continuous reliability improvements
- Build tooling, monitoring and logging systems that help hardware engineers triage and root-cause issues across different hardware platforms
- Create automated validation and characterization workflows for GPU, CPU, and high-performance compute platforms, including result collection, analysis, and reporting
- Build and operate data pipelines, dashboards, and analytics that surface hardware failure trends, performance regressions, and reliability metrics
- Support E2E integration and solution testing by developing test harnesses and CI/CD workflows that ensure Crusoe Cloud products meet performance, reliability, and scalability expectations
- Evaluate and introduce new tools, testing methods, and automation approaches that improve engineering efficiency and support Crusoe’s sustainability goals
- Provide data-driven insights from tooling and telemetry to influence Crusoe’s hardware roadmap and reliability strategy
Benefits
- Health & wellbeing: Comprehensive health benefits designed to support your overall wellness
- Time away: Paid time off for vacations, family bonding, and unexpected needs
- 401(k) match: Build your financial future with our 401(k) matching program
- Mental wellness: Resources and support for your emotional wellbeing and navigating life’s challenges
- Experience with BMC/Redfish/IPMI, firmware update automation, or out-of-band management
- Excellent communication and collaboration skills, with the ability to gather requirements from engineers and external partners and translate them into usable tools
- Experience working across hardware, firmware, and software teams in multidisciplinary environments
- Experience designing and implementing automation and test frameworks, including workflow engines, for hardware or large-scale infrastructure
- Strong proficiency in Python and at least one systems language (Go, C/C++, or Rust), plus Shell scripting
- Bachelor’s or Master’s degree in Computer Science, Computer Engineering, Electrical Engineering, or equivalent experience
- Experience building data pipelines, logging, telemetry, or observability tooling to analyze system behavior at scale
- Experience with fleet management, provisioning, or large-scale automation tools (Kubernetes, Ansible, Terraform, etc.)
- 5+ years of professional software engineering experience, ideally building internal tools, infrastructure software, provisioning, test automation, or systems software
- Strong analytical and problem-solving skills with a data-driven approach
- Experience building tooling for GPU clusters, including GPU-to-GPU communication diagnostics (NCCL tests, NVLink monitoring, etc.)
- Familiarity with NVIDIA tooling such as DCGM, NVML, and fabric management software
- Working knowledge of Linux systems, including kernel interfaces, drivers, device tooling, and low-level system diagnostics
- Familiarity with PCIe, InfiniBand, and NVMe/storage concepts and tooling (e.g., lspci, ibstat, nvme-cli, perftest, fio), with the curiosity to go deeper
- Background in factory test, manufacturing test software, or hardware qualification workflows
- Experience supporting both ARM and x86 server architectures
Skills
Sign in to read and join the discussion.