Agent Observability Engineer Jobs

1 job found

Today

AI Engineer - Data and Evals

Glean Technologies San Francisco Bay Area, CA

San Francisco Bay Area, CA

Today

$200k - $280k/yr Full time Agent Observability Ai Evals Artificial Intelligence

Apply Now

Job Description

Glean Technologies is building AI products and agents that teams can trust, and this role helps make reliability measurable. As an AI Engineer - Data and Evals, you will create evaluation datasets and infrastructure, run large-scale tests, and use those results to improve real agent behavior, including correctness, groundedness, citation quality, and task success.

This is a hybrid position in the San Francisco Bay Area, CA, based on working 4 days a week in the Mountain View or San Francisco office.

What you’ll do

Develop evalsets and benchmarks using expert-curated examples, production data, synthetic data, and prior failures, with coverage for edge cases and key user slices.
Build and scale an eval platform and agent observability, including experiment pipelines, trace capture and replay, and dashboards, with reproducible and affordable runs.
Create agentic LLM verifiers for correctness, groundedness, citation quality, and task success, then calibrate them against human labels using clear rubrics.
Run evals across the lifecycle, from CI and pre-release suites to sampled production traffic, and share outcomes using confidence intervals.
Build enterprise simulation environments that use simulated users and realistic company data to test multi-step agent tasks.
Investigate failures by slice and trace, translate evaluation results into concrete fixes, and balance quality with latency and cost tradeoffs.

What you bring

5+ years of software engineering experience with strong Go and/or Python and backend fundamentals, including APIs, data modeling, testing, and distributed pipelines. TypeScript is a plus.
Hands-on experience building datasets and benchmarks for LLM use cases, RAG, or agents, including evalset curation and synthetic data generation.
Familiarity with eval methodology tradeoffs such as deterministic metrics, LLM-as-a-judge, pointwise vs pairwise grading, human labeling, and judge calibration.
Experience building eval or experiment infrastructure that runs reliably at scale, including CI/CD integrations and observability tooling like OpenTelemetry.
Strong analytical skills: investigating traces and failure cases, writing SQL, and explaining quality movement using data and metrics.
Nice to have: experience with simulation environments and/or model post-training techniques.

Tools and technologies

Go, Python, TypeScript
SQL, OpenTelemetry
CI/CD

Compensation and benefits

Salary range: USD 200,000 - 280,000 per year.
Compensation depends on location, level, and relevant knowledge, skills, and experience.
Certain roles may be eligible for variable compensation, equity, and benefits.
Medical, Vision, and Dental coverage.
Generous time-off policy.
401(k) support opportunity to contribute to your plan.
Home office improvement stipend.
Annual education and wellness stipends.
Healthy lunches daily.

AI-first interview focus

During the interview process, you will complete a brief AI-focused exercise or discussion to show how you think about, design, and apply AI to drive impact in your role.

View full job page →

Get Agent Observability Engineer Jobs alerts

New Agent Observability Engineer Jobs, daily.