Senior Software Engineer, Production Pipelines

The role

This is a production engineering role. You will co-design and productionize the models our team trains and run them reliably and efficiently across very large image volumes. Your work will include systems engineering and innovations in the machine learning pipelines and infrastructure: packaging, scheduling, distributed inference, data movement, monitoring, and an automated loop to send model errors or failures back to annotation and retraining.

You report to the Head of Computation, and work, as needed, with other Mindspan teams (Microscopy and Optics, Expansion Microscopy, Scalable Chemistry, and Automation and Industrialization), using logic and evidence to advance your work so as to boost the performance of all.

The position is intended for an experienced software engineer from industry who has run machine learning in production and wants to apply that experience to science. Research experience is not required. What matters is that you have owned a system other people depended on, measured it, and made it faster and cheaper over time. The job also requires systematic documentation, analysis, and iterative problem solving, to arrive at working pipelines, apply them, and share them.

What you'll own

• Turn trained research models into versioned, tested, deployable inference services and batch jobs.
• Build and run distributed inference over very large, chunked volumes
• Own the automated training loop
• Build the evaluation and release process, so that every model version is scored on the same benchmarks before it runs on production data
• Instrument the pipeline: throughput, GPU use, cost per unit volume, failure rates, and data lineage from raw image to final output.
• Make the system usable by scientists
• Documentation and training: keep runbooks and design records that others can execute, with the reasoning behind each decision

Culture we're building

The disorders of the brain affect over 1,000,000,000 people around the world: Alzheimer’s, Parkinson’s, stroke, and other conditions. We operate with a sense of urgency, even as we look to understand the brain at its most fundamental levels.

We are hyper collaborative and focused on impact over short-term markers of productivity. Creativity and failure are the steppingstones to scale, and working together with “strong opinions, weakly held” is our style.

We solve real engineering problems the best way we can: inventing fast when we have to, and using good existing solutions where they exist. We believe hard problems only yield to many disciplines working together, so collaboration is not optional.

We expect people to explore the unknown, which means making mistakes daily, owning them, and learning from them to improve and excel.

We are recruiting a team of scientists and engineers who exemplify all these values.

What we're looking for

• Turn trained research models into versioned, tested, deployable inference services and batch jobs.
• Build and run distributed inference over very large, chunked volumes
• Own the automated training loop
• Build the evaluation and release process, so that every model version is scored on the same benchmarks before it runs on production data
• Instrument the pipeline: throughput, GPU use, cost per unit volume, failure rates, and data lineage from raw image to final output.
• Make the system usable by scientists
• Documentation and training: keep runbooks and design records that others can execute, with the reasoning behind each decision

What we offer

The chance to build the expansion and staining pipeline of a new research institute alongside some of the most adventurous scientists in the field, with real autonomy and a direct line to the people setting the scientific agenda. You will get to contribute to addressing one of the most important unmet medical needs of our time, as part of the initial team. You will learn lots of things that probably cannot be learned any other way, as you solve problems, build, and work with others. As Mindspan grows, we expect this role to grow with it. We are excited to invest in someone who wants to take on increasing responsibility over time.

Comprehensive health, dental, vision, and family benefits, and a matching retirement contribution.

Competitive salary ($180,000 - $235,000), depending on experience.