Evaluation And Metrics Engineer Jobs

1 job found

Today

Staff Software AI Engineer

Rivian Palo Alto, CA

Palo Alto, CA

Today

Full time Backend Developer Agentic Ai Platform Artificial Intelligence

Apply Now

Job Description

Rivian is building reliable internal AI capabilities through its Applied AI team. In this onsite Staff Software AI Engineer role in Palo Alto, you will help shape the technical direction for integrating large language models with enterprise knowledge and tools to create agentic applications that can operate at scale.

What you will deliver

Own the agent platform architecture, including orchestration for long-running, multi-step agents; persistent state management; tool integration; permission-aware access to enterprise data; and evaluation infrastructure that gates production releases.
Build reusable patterns and standards so capabilities created for one application can be adopted across the organization with consistent interfaces and engineering practices.
Set the bar for reliability, observability, and evaluation, ensuring production systems meet defined quality, latency, and cost targets against established baselines.
Lead complex engineering by designing and implementing the hardest components, delivering reference implementations, and resolving critical integration and reliability challenges.
Guide technical decisions on models and tools by evaluating emerging models, frameworks, and tools; choosing build versus buy and vendor selections; and translating advances into a roadmap with measurable impact.
Partner on governance and control with Security, Legal, and Compliance to embed authorization, auditability, and data governance into the platform and its applications.
Raise the engineering bar through design and code reviews, mentoring, and participation in hiring.

What you bring

8+ years of professional software engineering experience, including technical leadership of complex production systems.
Proven experience shipping and operating LLM-based or agentic applications with real users in production, supported by measured quality, adoption, and business impact.
Deep expertise in agent system design: orchestration, tool calling, retrieval and grounding, context and memory management, durable execution, verification, and evaluation of nondeterministic systems.
Backend and distributed systems expertise: APIs, asynchronous execution, state management, fault tolerance, and observability at scale.
Expert-level Python and experience deploying services on cloud infrastructure using containers and automated delivery pipelines.
A track record of setting technical direction and engineering practices that were adopted beyond your team, with demonstrable impact.
Experience integrating with enterprise systems and data under security, privacy, and permission constraints.
Ability to turn ambiguous needs into practical solutions and communicate clearly with engineers, domain experts, and executives.
Degree in Computer Science, Engineering, or a related field (Bachelor’s, Master’s, or Ph.D.), or equivalent practical experience.

Technologies

Python
Containers

Preferred qualifications

Experience with multi-agent systems including verification or adversarial checking, persistent state, and recovery or human-approval mechanisms.
Experience with multimodal applications, including document or engineering-drawing understanding, vision models, or computer-use agents.
Familiarity with product development tooling such as CAD, PLM, simulation or CAE, or physics-informed machine learning.
Experience improving AI systems through failure analysis, feedback loops, model adaptation, or fine-tuning.
Experience building AI infrastructure or capabilities adopted by multiple teams, including forward-deployed work at an AI lab or agent-infrastructure company.

View full job page →

Get Evaluation And Metrics Engineer Jobs alerts

New Evaluation And Metrics Engineer Jobs, daily.