Senior Software Engineer (Agentic Intelligence)

• AI agents at Honeycomb investigate, reason, and act on real observability data. They live in Canvas: the agentic workspace where engineers go to understand their systems
• The Agentic Intelligence team has shipped Canvas, the Honeycomb MCP server, and the Canvas Agent and Canvas Skills surfaces
• Honeycomb’s data store is fast and accepts high cardinality data; that’s what makes agents built on top of it different from anything built on a conventional observability backend
• This role is about taking advantage of that building agents that can do things no other observability product can do because the underlying data makes it possible
• Some of this work will start as a prototype. The expectation is that the code makes it through the full arc, from the rough first version through to something that holds up in production
• Design and deliver production-grade agents. Build agents that investigate, reason, and act on live observability data inside Canvas. These agents must be trustworthy to engineers in high pressure situations, including mid-incident. Take one from rough first version to something that holds up under production traffic
• Own the agent work; support the whole product. Scope, build, ship, and maintain the agents including the evals that tell you whether they got better or are just different. This role is agent-focused and also includes some fullstack development
• Build agents only Honeycomb can build. Use a data store that returns high-cardinality queries in seconds to reason over signal a conventional backend can’t serve at this fidelity correlating across services, drilling into a single trace, comparing before and after a deploy
• Extend the surface, and decide what’s next. Ship new capability into Canvas, the MCP server, and Canvas Skills memory, spatial awareness, a faster Bedrock loop and make the case for what comes after with working code. Distinguish hype from signal in a field with plenty of both
• Define what “good” means for agents here. Set the bar: measurable against real evals, maintainable, and honest about their limits
• Example projects:
• Multiple agents collaborating on one shared Canvas investigation each claiming a hypothesis, publishing findings, and narrowing the search space for the others so it resolves faster (blog)
• Auto-investigation the moment an SLO burn alert fires the agent forms hypotheses and prepares visualizations before a human looks, cutting mean-time-to-insight for on-call (o11ycon 2026)
• Skills that encode a team’s domain expertise e.g. Kubernetes thresholds so agents and human colleagues can lean on them (o11ycon 2026)

Benefits

• 100% covered medical, dental, vision, HSA, FSA, disability, and more
• Unlimited PTO, synchronous company break, paid sabbatical, and expressed minimum for time to recharge
• 401K retirement plans and fully paid life insurance
• Up to 16 weeks paid leave regardless of path to parenthood; flexible hours for returning parents and caregivers
• $10K to cover gaps in insurance, in support of adoption, surrogacy, preservation, and family-planning costs
• $600 per year to spend on the perks you care for most
• A stake in our success
• Generous new hire equity, and refresh grants
• Annual $1,500 development stipend; funding for cognitive coaching; and a focus on career pathing
• $500 new hire home setup stipend and $200/mo for wifi, phone and co-working

• What we’re looking for today is someone who brings deep agent expertise and uses it to expand what the team can build: new agents, new surface area in Canvas, memory, spatial awareness, improved performance on our Bedrock loop
• Product judgment. You can look at what the agent layer does today and see what it should do next and make that case with a prototype, not a deck
• Agent architecture depth. You understand how a fast, high-cardinality data store changes what an agent can reason about, and how to design for that
• End-to-end ownership. On a small team there’s no handoff queue. You can take something from rough prototype to production-grade without needing someone behind you to do the durable engineering
• Current judgment, not just past experience. You have informed opinions about what’s shifted in agent design in the last six to twelve months that would change how you’d build today
• AI and agent engineering experience. You’ve shipped LLM-based systems people relied on in production not demos, not fine-tuned models in a research context. You know where agent systems break and how to design around it
• Observability or developer-tools background. Engineers are your users; you’ll ramp faster with fluency in that world, and the work is better
• Familiarity with eval frameworks, agent tooling, RAG, and prompt engineering