FurtherAI is building AI agents for real insurance workflows, and this role owns the evaluation and model quality layer that helps teams ship improvements safely. You will help turn agent traces, customer feedback, and production outcomes into measurable benchmarks, automate quality checks across the platform, and continuously improve reliability across assistant, document processing, and computer use.
This is an onsite position in San Francisco, CA with fully covered health, dental, and vision, competitive compensation ranging from $300,000 to $350,000, meaningful stock options, and company 401(k). Enjoy unlimited PTO, lunch and dinner onsite, and an outstanding in-office culture with team events including happy hours and off-sites (5x per week). Pre-tax commuter benefits are also included.
What you’ll do
- Own evaluations across FurtherAI: Build the evaluation frameworks and infrastructure to measure accuracy, quality, and reliability across workflows, Assistant, document processing, and computer use.
- Build high-quality datasets: Create golden datasets, benchmarks, and regression suites with Forward Deployed Engineers and domain experts to cover real-world insurance tasks and edge cases.
- Evaluate complex agent behavior: Design evaluations for multi-step trajectories, tool use, reasoning, and end-to-end task completion.
- Build automated grading systems: Develop LLM-as-judge evaluations, deterministic graders, human evaluation workflows, and rubrics aligned with real-world quality.
- Make development evaluation-driven: Integrate offline and online evaluations into development and release processes to measure improvements, catch regressions, and support confident shipping.
- Drive model evaluation and selection: Continuously benchmark frontier and open-weight models for accuracy, latency, cost, and reliability to guide model selection, upgrades, and production releases.
- Build continuous improvement loops: Convert production failures, customer feedback, and agent traces into new evaluation cases and measurable gains in agent quality.
What you bring
- 5+ years of software engineering experience with hands-on experience building production AI/ML systems.
- Experience building evaluation frameworks, LLM-based applications, model benchmarking systems, or applied AI infrastructure.
- Strong understanding of modern LLMs, agent architectures, evaluation methodologies, and model performance tradeoffs.
- Python engineering skills for reliable, production-grade systems.
- Ability to independently own ambiguous technical problems from research and experimentation through production.
- Clear communication in person and in writing.
- Bonus: Experience with LLM-as-judge, human evaluation systems, agent trajectory evaluations, or large-scale benchmarking.
- Bonus: Experience with open-weight models, fine-tuning, post-training, or model routing.
- Bonus: Background in B2B SaaS and 0-to-1 experience.
- Above all, drive, grit, and ownership.
Representative projects
- Build an end-to-end evaluation framework for FurtherAI agents, measuring success across multi-step workflows, tool calls, and complex insurance tasks.
- Create a reusable library of golden datasets and benchmarks for common insurance workflows across customer implementations and out-of-the-box AI skills.
- Develop an automated benchmarking system to evaluate new frontier and open-weight model releases and identify best models per task.
- Implement production evaluation pipelines to detect quality regressions, identify failure patterns, and turn them into actionable improvements.
Technologies: Python, LLMs, open-weight models, fine-tuning, post-training, model routing
