Agent Evaluation Engineer Jobs

1 job found

8d ago

Member of Technical Staff, AI Engineer

Parameter ai San Francisco, CA

San Francisco, CA

8d ago

$180k - $240k/yr Full time Backend Developer Agent Evaluation Agentic Ai

Apply Now

Job Description

Parameter ai is looking for a Member of Technical Staff, AI Engineer in San Francisco to develop LangGraph and Temporal-based agent systems for autonomous offensive security testing. The work centers on building exploitation workflow automation, tool interfaces, and evaluation loops that help reduce false positives while keeping inference cost and latency under control.

This onsite role focuses on taking agent behavior from design to reliable production execution, with an emphasis on verification and measuring outcomes so agent runs improve over time.

Responsibilities

Design and ship agent workflows in LangGraph and Temporal that plan, execute, and verify multi-step exploitation chains
Build the agent tool layer, including HTTP clients, auth handling, browser control, cloud enumeration, and payload generation
Own the eval loop using Braintrust and Langfuse to define success criteria for agent runs and drive improvements
Reduce false positives, identified as the hardest challenge in this area and a key customer priority
Manage inference cost and latency at scale, including prompt caching, context strategy, and model selection across Claude and Bedrock

Requirements

3+ years of software engineering, with at least one year building production LLM systems (not limited to prototypes or demos)
Strong TypeScript
Comfort working with async, distributed systems, and long-running workflow orchestration
Intuitive grasp of context engineering, tool design, and why agents fail outside controlled environments
A measurement mindset: built or maintained an eval harness and understands why offline metrics can diverge from production
Bonus: security background, CTF experience, bug bounty history, or a track record of breaking things you were not supposed to break

Technologies

TypeScript, LangGraph, Temporal, GCP
AWS Bedrock, Anthropic API
Langfuse, Braintrust
GitHub Actions, Linear, Graphite

Interview Process

Intro call with a founder (30 minutes)
Technical deep dive on your past work (60 minutes)
Paid work trial or take-home, scoped to roughly one day
Onsite in San Francisco with the team
References and offer

Location: San Francisco, CA (onsite). Compensation: USD 180,000 - 240,000 per year. Work authorization: You must be authorized to work in the US; Parameter ai is not able to sponsor visas at this time.

View full job page →

Get Agent Evaluation Engineer Jobs alerts

New Agent Evaluation Engineer Jobs, daily.