Senior Software Engineer II - Agentic Intelligence
Job Description
Honeycomb is hiring a Senior Software Engineer II to build production-grade AI agents for live observability analysis inside Canvas.
Responsibilities
- Design and deliver production-grade agents that investigate, reason, and act on live observability data inside Canvas.
- Own agent delivery end-to-end: scope, build, ship, and maintain agents, including evals to confirm agents improve or meaningfully change behavior.
- Build agent capabilities enabled by a data store that returns high-cardinality queries in seconds, so agents can correlate across services, drill into a single trace, and compare before/after deploys.
- Extend Canvas and adjacent surfaces by shipping new capabilities into Canvas, the MCP server, and Canvas Skills memory.
- Improve agent performance and iteration speed with a faster Bedrock loop, then propose what comes next with working code.
- Set and uphold a bar for “good” agents using real evals, focusing on measurable quality, maintainability, and clarity about agent limits.
Requirements
- AI and agent engineering experience, including shipping LLM-based systems relied on in production (not demo-only) and understanding where agent systems break and how to design around it.
- End-to-end ownership: take work from rough prototype through production-grade durable engineering without needing someone behind you to drive reliability.
- Current judgment: informed opinions about changes in agent design over the last six to twelve months that would affect how you build today.
- Agent architecture depth, including how a fast high-cardinality data store changes what agents can reason about and how to architect for it.
- Product judgment: identify what the agent layer should do next and make the case with a prototype rather than a deck.
Technologies
- LLM-based systems
- RAG
About the Role
- AI agents investigate, reason, and act on real observability data inside Canvas, the agentic workspace where engineers understand their systems.
- The Agentic Intelligence team has shipped Canvas, the Honeycomb MCP server, and the Canvas Agent and Canvas Skills surfaces.
- Some work may begin as a prototype, with expectations to carry code through the full arc to production-ready behavior.
Example Projects
- Multiple agents collaborating on a shared Canvas investigation, each proposing a hypothesis, publishing findings, and narrowing the search space to resolve faster (blog).
- Auto-investigation when an SLO burn alert fires: the agent forms hypotheses and prepares visualizations before a human looks, reducing mean-time-to-insight for on-call (o11ycon 2026).
- Skills encoding team domain expertise (for example, Kubernetes thresholds) so agents and human colleagues can reuse that knowledge (o11ycon 2026).
Even Better
- Observability or developer-tools background.
- Familiarity with eval frameworks, agent tooling, RAG, and prompt engineering.
Benefits
- Generous equity with an employee-friendly stock program
- Unlimited PTO
- Distributed-first mindset and culture
- Home office, co-working, and internet stipend
- Full benefits coverage for employees, with additional coverage available for dependents
- Up to 16 weeks of paid parental leave, regardless of path to parenthood
- Annual development allowance
Location
- Remote (remote)
Compensation
- Base salary: $183,340 - $206,000 USD (based on level of experience)
Important Notices
- Honeycomb cannot currently sponsor or support visa transfers at this time. In compliance with applicable law, all persons hired will be required to verify identity and eligibility to work.
- Phishing warning: All communications will come from an @honeycomb.io email address. The recruiting process will never ask for financial or sensitive personal information such as Social Security or tax identification numbers, credit card numbers, or bank account information.
- Diversity & accommodations: Honeycomb is committed to building a diverse, inclusive, and equitable workplace and will provide reasonable accommodations during the hiring process.
- Privacy notice: If an application is unsuccessful (or you withdraw/decline), Honeycomb will retain your information for a period of time in accordance with local laws. For more information, see the Honeycomb Privacy Notice; contact [email protected] to opt out of retention for other roles or to update your information.