AI Engineer - Data and Evals

Glean Technologies is building AI products and agents that teams can trust, and this role helps make reliability measurable. As an AI Engineer - Data and Evals, you will create evaluation datasets and infrastructure, run large-scale tests, and use those results to improve real agent behavior, including correctness, groundedness, citation quality, and task success.

This is a hybrid position in the San Francisco Bay Area, CA, based on working 4 days a week in the Mountain View or San Francisco office.

What you’ll do

• Develop evalsets and benchmarks using expert-curated examples, production data, synthetic data, and prior failures, with coverage for edge cases and key user slices.
• Build and scale an eval platform and agent observability, including experiment pipelines, trace capture and replay, and dashboards, with reproducible and affordable runs.
• Create agentic LLM verifiers for correctness, groundedness, citation quality, and task success, then calibrate them against human labels using clear rubrics.
• Run evals across the lifecycle, from CI and pre-release suites to sampled production traffic, and share outcomes using confidence intervals.
• Build enterprise simulation environments that use simulated users and realistic company data to test multi-step agent tasks.
• Investigate failures by slice and trace, translate evaluation results into concrete fixes, and balance quality with latency and cost tradeoffs.

What you bring

• 5+ years of software engineering experience with strong Go and/or Python and backend fundamentals, including APIs, data modeling, testing, and distributed pipelines. TypeScript is a plus.
• Hands-on experience building datasets and benchmarks for LLM use cases, RAG, or agents, including evalset curation and synthetic data generation.
• Familiarity with eval methodology tradeoffs such as deterministic metrics, LLM-as-a-judge, pointwise vs pairwise grading, human labeling, and judge calibration.
• Experience building eval or experiment infrastructure that runs reliably at scale, including CI/CD integrations and observability tooling like OpenTelemetry.
• Strong analytical skills: investigating traces and failure cases, writing SQL, and explaining quality movement using data and metrics.
• Nice to have: experience with simulation environments and/or model post-training techniques.

Tools and technologies

• Go, Python, TypeScript
• SQL, OpenTelemetry
• CI/CD

Compensation and benefits

• Salary range: USD 200,000 - 280,000 per year.
• Compensation depends on location, level, and relevant knowledge, skills, and experience.
• Certain roles may be eligible for variable compensation, equity, and benefits.
• Medical, Vision, and Dental coverage.
• Generous time-off policy.
• 401(k) support opportunity to contribute to your plan.
• Home office improvement stipend.
• Annual education and wellness stipends.
• Healthy lunches daily.

AI-first interview focus

During the interview process, you will complete a brief AI-focused exercise or discussion to show how you think about, design, and apply AI to drive impact in your role.