Software Engineer II - Compute
Posted Updated
Description
You will design, implement, and maintain core components of a high-performance distributed batch compute engine, including its scheduler, resource allocator, and execution engine. You will build low-latency APIs connecting a Python SDK to a Golang core, develop workflow abstractions, solve distributed locking and fair-share scheduling problems, and collaborate with engineering teams to integrate compute workflows.
Responsibilities
- Design, implement, and maintain core components of the distributed batch compute engine (BatchAPI)
- Architect and optimize the scheduler, resource allocator, and execution engine
- Design low-latency APIs and resilient communication protocols bridging the Python SDK and Golang core
- Develop high-level workflow abstractions for data processing, simulation, and ML training pipelines
- Solve distributed locking, throttling, and fair-share scheduling problems
- Improve performance, scalability, and resilience, implementing monitoring and alerting
- Collaborate with infrastructure and product engineering teams to gather requirements and integrate compute workflows
Requirements
- 5+ years of professional software engineering experience
- Deep expertise in Golang and some knowledge of Python
- Strong understanding of distributed systems fundamentals
- Experience with performance profiling and tuning
- Specialized knowledge of Kubernetes
- Proven track record of driving performance, scalability, and resilience improvements in production
- Familiarity with cloud services (AWS EKS, S3, RDS)
Benefits
- Annual bonus
- Equity compensation
- Benefits
Skills
Sign in to read and join the discussion.