Senior Software Engineer (Crusoe Managed Data Transfer)
Posted Updated
Description
- Crusoe’s Managed Platform Services (MAPS) organization builds the managed services that make Crusoe Cloud easy to adopt and hard to leave, including Crusoe Container Registry, Cloud Monitoring and Managed Logs
- Managed Data Transfer (MDT) is its newest service
- MDT moves petabyte-scale training data from AWS S3, and later other clouds, into Crusoe storage with a single API call: the service shards the work across a fleet of transfer workers, verifies every object, and asks the customer to run nothing
- For AI labs and Physical AI companies whose data lives with a hyperscaler, it removes one of the biggest obstacles to training on Crusoe GPUs
- We are hiring Senior Software Engineers to build the MDT control plane: the job API, scheduler and job state that turn a customer request into a coordinated, resumable transfer
- You will join a new team at the beginning, take the service from first commit to its preview launch with design-partner customers, and shape the API that new clouds and Crusoe-internal transfers will plug into later
- You will work closely with the Storage, Network, IAM and Cloud Console teams, and report to the Engineering Manager of Managed Data Transfer
- Job API and Lifecycle: Design and build the data-transfer API (create, list, get, cancel) with a provider-neutral model for sources and destinations, so new clouds and storage types can be added without breaking customers
- Scheduling and Job State: Build the scheduler and state machine that split a bucket listing into units of work, place them on regional worker pools, and track progress, retries and resume across failures
- Correctness at Scale: Make petabyte-scale jobs safe to retry and cancel, with end-to-end checksum verification and clear errors when a customer’s permissions or destination are wrong
- Cross-Cloud Identity: Implement secure access to customer buckets with short-lived, least-privilege credentials (AWS role assumption first, other clouds next), integrated with Crusoe IAM and audit logging
- Customer Surfaces: Ship the CLI and, with partner teams, the Terraform and Console experiences for creating and monitoring transfers
- Operational Excellence: Instrument the service with metrics, logs and alerts, write the runbooks, and take part in the team’s on-call rotation
- Cross-Team Collaboration: Agree the interfaces MDT depends on with Storage, Network, IAM and Console, and work with Product and Solutions Architects on early customer onboarding
Benefits
- Health & wellbeing: Comprehensive health benefits designed to support your overall wellness
- Time away: Paid time off for vacations, family bonding, and unexpected needs
- 401(k) match: Build your financial future with our 401(k) matching program
- Mental wellness: Resources and support for your emotional wellbeing and navigating life’s challenges
- Educational Foundation: A bachelor’s degree in Computer Science, Engineering, or a related field, or equivalent professional experience
- Backend Engineering Craft: Experience designing and shipping production backend services and APIs, ideally in Go, with gRPC or REST, relational databases and Kubernetes
- Collaboration and Writing: You work well across teams with different priorities, and your design docs settle decisions
- Security Mindset: Working knowledge of authentication, authorization and least privilege, and care in handling customer credentials
- API Design: Ability to design public APIs that customers automate against and that can grow without breaking changes
- Distributed Systems Judgment: Comfort with long-running, failure-prone work: idempotency, retries with backoff, leases, checkpointing and partial failure
- Professional Experience: 5+ years building cloud infrastructure or platform services, including taking a service from zero to general availability
- Operational Ownership: You have run services in production, been on call, and built the dashboards and alerts that keep incidents short
- Experience building on GPU or AI infrastructure
- Experience with job orchestration systems such as Temporal, Argo, or Kubernetes operators
- Experience with storage systems (object storage, NFS, parallel file systems) or high-throughput networking
- Hands-on experience with cross-cloud identity: AWS IAM and STS, GCP service-account impersonation, or Azure workload identity
- Experience with large-scale data movement, such as S3 multipart uploads, rclone, s5cmd, AWS DataSync or Google Storage Transfer Service
Skills
Sign in to read and join the discussion.