Senior Infrastructure Engineer
Posted Updated
Description
- Software Infrastructure runs the Kubernetes clusters, CI/CD systems, and compute environments that every d-Matrix engineer builds and tests on, along with the systems behind our externally accessible product demos
- As a senior infrastructure engineer, you will be the team’s first responder and own issues end to end: diagnosing them, fixing them, and engaging IT or the product software group when the root cause sits with them
- Our environment combines cloud resources, conventional servers, and our own AI accelerators and networking hardware, so quickly isolating the cause is central to the job
- When an issue recurs, you will turn it into automation, better alerting, or a runbook
- This is not an IT support or ticket-queue role, and IT owns physical hardware; we manage everything from the operating system up
- On-call participation is expected
- Own first response and resolution for issues across Kubernetes clusters, GitLab and CI runners, VMs, standalone systems, and demo environments
- Triage across teams with IT and the product software group when the root cause sits on their side
- Convert recurring issues into automation, alerts, and runbooks so repeat incidents decline
- Improve monitoring and alerting to reduce false alarms and speed detection of real problems
- Perform routine operational work, including upgrades, patching, certificate renewals, and capacity checks
- Ship automation and improvements the team relies on, beyond individual fixes
- Participate in on-call coverage for the environment
- Contribute to design and build work across the team’s systems as your scope grows
Benefits
- Equity
- Health care
- Flexible time-off
- Paid paternity leave
- 401k retirement plan
- Remote & hybrid working model
- Work with world-leading engineers and researchers
- On-site fitness center
- Team-building activities
- Free food and snacks
- Peer coaching & development
- Happy hours
- Drive and curiosity to learn unfamiliar systems and how state-of-the-art inference systems are built
- Working knowledge of networking fundamentals, including DNS, routing, load balancing, and TLS
- Clear communication under pressure, including concise incident status updates
- 2+ years operating production Linux systems, including hands-on Kubernetes troubleshooting
- Methodical debugging skills: forming hypotheses, narrowing them quickly, and working across layers when the obvious answer is wrong
- Scripting experience in Python and/or Bash
- Experience with monitoring and observability tools
- Familiarity with CI/CD systems (GitLab preferred)
- Experience with GPU or accelerator nodes in Kubernetes
- Experience with infrastructure-as-code tools such as Ansible or Terraform
- Experience with monitoring platforms such as DataDog, Prometheus, or Grafana
- Bare metal or on-prem infrastructure experience
- CKA or CKAD certification
Skills
Sign in to read and join the discussion.