EngineerJobs.io
← Back to all jobs
Honeycomb.io

Senior Software Engineer II - Agentic Intelligence

Remote Remote $183k - $206k/yr Full time Posted 2h ago

Job Description

Honeycomb is hiring a Senior Software Engineer II to build production-grade AI agents for live observability analysis inside Canvas.

Responsibilities

  • Design and deliver production-grade agents that investigate, reason, and act on live observability data inside Canvas.
  • Own agent delivery end-to-end: scope, build, ship, and maintain agents, including evals to confirm agents improve or meaningfully change behavior.
  • Build agent capabilities enabled by a data store that returns high-cardinality queries in seconds, so agents can correlate across services, drill into a single trace, and compare before/after deploys.
  • Extend Canvas and adjacent surfaces by shipping new capabilities into Canvas, the MCP server, and Canvas Skills memory.
  • Improve agent performance and iteration speed with a faster Bedrock loop, then propose what comes next with working code.
  • Set and uphold a bar for “good” agents using real evals, focusing on measurable quality, maintainability, and clarity about agent limits.

Requirements

  • AI and agent engineering experience, including shipping LLM-based systems relied on in production (not demo-only) and understanding where agent systems break and how to design around it.
  • End-to-end ownership: take work from rough prototype through production-grade durable engineering without needing someone behind you to drive reliability.
  • Current judgment: informed opinions about changes in agent design over the last six to twelve months that would affect how you build today.
  • Agent architecture depth, including how a fast high-cardinality data store changes what agents can reason about and how to architect for it.
  • Product judgment: identify what the agent layer should do next and make the case with a prototype rather than a deck.

Technologies

  • LLM-based systems
  • RAG

About the Role

  • AI agents investigate, reason, and act on real observability data inside Canvas, the agentic workspace where engineers understand their systems.
  • The Agentic Intelligence team has shipped Canvas, the Honeycomb MCP server, and the Canvas Agent and Canvas Skills surfaces.
  • Some work may begin as a prototype, with expectations to carry code through the full arc to production-ready behavior.

Example Projects

  • Multiple agents collaborating on a shared Canvas investigation, each proposing a hypothesis, publishing findings, and narrowing the search space to resolve faster (blog).
  • Auto-investigation when an SLO burn alert fires: the agent forms hypotheses and prepares visualizations before a human looks, reducing mean-time-to-insight for on-call (o11ycon 2026).
  • Skills encoding team domain expertise (for example, Kubernetes thresholds) so agents and human colleagues can reuse that knowledge (o11ycon 2026).

Even Better

  • Observability or developer-tools background.
  • Familiarity with eval frameworks, agent tooling, RAG, and prompt engineering.

Benefits

  • Generous equity with an employee-friendly stock program
  • Unlimited PTO
  • Distributed-first mindset and culture
  • Home office, co-working, and internet stipend
  • Full benefits coverage for employees, with additional coverage available for dependents
  • Up to 16 weeks of paid parental leave, regardless of path to parenthood
  • Annual development allowance

Location

  • Remote (remote)

Compensation

  • Base salary: $183,340 - $206,000 USD (based on level of experience)

Important Notices

  • Honeycomb cannot currently sponsor or support visa transfers at this time. In compliance with applicable law, all persons hired will be required to verify identity and eligibility to work.
  • Phishing warning: All communications will come from an @honeycomb.io email address. The recruiting process will never ask for financial or sensitive personal information such as Social Security or tax identification numbers, credit card numbers, or bank account information.
  • Diversity & accommodations: Honeycomb is committed to building a diverse, inclusive, and equitable workplace and will provide reasonable accommodations during the hiring process.
  • Privacy notice: If an application is unsuccessful (or you withdraw/decline), Honeycomb will retain your information for a period of time in accordance with local laws. For more information, see the Honeycomb Privacy Notice; contact [email protected] to opt out of retention for other roles or to update your information.

Similar Jobs