Evaluation And Evals Engineer Jobs

1 job found

Today

Staff Applied AI Engineer

Qualified Health PBC Palo Alto, CA

Palo Alto, CA

Today

$200k - $260k/yr Full time Agentic Ai Ai Agent Ai Agent Engineering

Apply Now

Job Description

Qualified Health PBC is hiring a Staff Applied AI Engineer in Palo Alto, CA (onsite) to lead technical ownership for agentic workflow infrastructure across product lines.

Responsibilities

Architect and build core agentic workflow infrastructure supporting 100+ workflows across multiple product lines.
Define team-wide technical standards and reusable patterns for agentic workflow development, including eval pipelines, prompt frameworks, and deployment patterns.
Lead design and code review for complex, cross-cutting integrations and production services.
Own evaluation and adoption decisions for state-of-the-art LLMs (OpenAI, Anthropic, Mistral, etc.) across use cases, including build-vs-buy tradeoffs.
Mentor engineers on production-grade agentic system design, MLOps, and shipping efficiently while maintaining reliability.
Partner with product, engineering, and design leadership to shape the technical roadmap for AI-powered features.
Identify and resolve systemic bottlenecks in workflow infrastructure before they limit broader team execution.

Requirements

Strong software engineering foundation; Python required and deep familiarity with production system design.
5+ years building production software, including 2+ years building AI-powered applications with LLMs (ChatGPT, Claude, Anthropic, Mistral, etc.).
Proven experience architecting REST APIs and services deployed at scale in cloud environments (GCP preferred).
Strong grasp of MLOps, model deployment, and workflow automation, including experience setting standards or infrastructure other engineers build on.
Track record of driving technical decisions with significant ambiguity and minimal oversight.

Technologies

Python
OpenAI
Anthropic
Mistral
ChatGPT
Claude
REST APIs
GCP
LLMs

Desirable skills

Bachelor’s + 7 years OR Master’s + 5 years in Computer Science, Engineering, or related field (or equivalent experience).
Background spanning both software engineering and ML/AI systems.
Experience designing eval pipelines, prompt tuning frameworks, or automation infrastructure at a platform level.
Startup experience with a bias toward shipping fast and iterating.
Exposure to both startup agility and the engineering rigor of a mature, at-scale codebase.
Experience mentoring engineers or setting technical direction for a team.

Pay & benefits

Salary range: $200,000 - $260,000 per year, depending on skills, qualifications, experience, and location.
Eligible for equity and benefits.
Competitive salaries with equity packages.
Robust medical/dental/vision insurance.
Flexible working hours.
Hybrid work options.
Inclusive environment that fosters creativity and innovation.

View full job page →

Get Evaluation And Evals Engineer Jobs alerts

New Evaluation And Evals Engineer Jobs, daily.