Llm Evaluation

Llm Evaluation Engineer Jobs

3 jobs found

13d ago

Machine Learning Engineer, Human Centered AI - Evaluations & Insights

Apple Seattle, WA

Seattle, WA

13d ago

$142k - $263k/yr Full time Adversarial Ai Evaluation Ai Evaluation Artificial Intelligence

Apply Now

Job Description

On Apple’s Human-Centered AI team, this role focuses on evaluating and improving Foundation Models and generative AI experiences with an emphasis on how people perceive quality. You will build evaluation frameworks and MLOps pipelines that turn model behavior into measurable insights, helping ensure AI systems are reliable, safe, and aligned with human expectations.

In this onsite position in Seattle, you will partner across engineering, research, product, and Responsible AI teams to refine model performance based on evaluation telemetry and human-centric measures.

Responsibilities

Architect and execute evaluation suites for LLMs and multimodal models, identifying edge cases across multi-step reasoning, factuality, adversarial robustness, safety, and alignment.
Build deterministic, heuristic, and LLM-assisted evaluation frameworks (including LLM-as-a-judge and reward modeling) to quantify human-perceived quality signals such as helpfulness and hallucination rates.
Convert qualitative failure modes into quantifiable loss patterns, programmatic guardrails, and actionable data-mixture adjustments for training and inference.
Collaborate with engineering teams to refine model behavior using evaluation telemetry to guide prompt engineering, Retrieval-Augmented Generation (RAG) strategies, and model fine-tuning.
Apply advanced ML techniques such as embedding-based clustering, representation learning, and perturbation analysis to map error taxonomies and latent failure manifolds.
Develop robust MLOps workflows to define evaluation metrics, automate regression testing across model checkpoints, and integrate human-centric assessments into ML CI/CD pipelines.
Architect scalable, distributed inference and processing pipelines (for example, Ray and vLLM) to support high-throughput evaluation, automated annotation, and output analysis.
Define quantitative evaluation frameworks that capture nuanced human factors, including trust calibration, conversational state tracking, and interpretability.
Build automated evaluation pipelines that use LLMs to assess outputs at scale and improve correlation with human baseline annotations.
Work with ML researchers, software developers, and product managers across Apple to translate product requirements into scalable, reliable, and efficient evaluation infrastructure.

Requirements

Knowledge of human factors, HCI, or cognitive science methodologies as applied to AI system design.
5+ years of relevant industry experience in ML Engineering or Applied Research.
Advanced proficiency in Python and modern deep learning ecosystems including PyTorch, JAX, and Hugging Face.
Experience building scalable ML inference pipelines, model evaluation workflows, and structured rating frameworks for large-scale AI systems.
Ability to interpret unstructured model outputs (text, transcripts, embedding spaces) and synthesize qualitative findings into actionable engineering guidance and training objectives.
Hands-on experience developing, fine-tuning, or evaluating LLMs, multimodal models, and NLP systems.
Familiarity with AI quality metrics, hallucination detection techniques such as SelfCheckGPT, model alignment methods including RLHF and DPO, and LLM-as-a-judge frameworks such as G-Eval and DeepEval.
Experience building internal tools or automated pipelines for ML workflows using MLflow, Weights & Biases, or similar platforms.
Strong familiarity with prompt engineering, RAG architectures (vector databases, semantic search), and fine-tuning.
Bachelor’s or Master’s degree in Computer Science, Machine Learning, Artificial Intelligence, Cognitive Science, or a related technical field.

Technologies

Python, PyTorch, JAX, Hugging Face
Ray, vLLM
MLflow, Weights & Biases
SelfCheckGPT, RLHF, DPO, G-Eval, DeepEval
LLM-as-a-judge, Retrieval-Augmented Generation (RAG)
Vector databases, semantic search

Benefits

Comprehensive medical and dental coverage.
Retirement benefits.
Range of discounted products and free services.
Reimbursement for certain educational expenses, including tuition, for formal education related to advancing your career.
Opportunity to become an Apple shareholder through participation in Apple’s discretionary employee stock programs.
Eligible for discretionary restricted stock unit awards.
Option to purchase Apple stock at a discount if voluntarily participating in Apple’s Employee Stock Purchase Plan.
May be eligible for discretionary bonuses or commission payments and relocation.

View full job page →

1mo ago

Machine Learning Engineer, GAI Search Platform

ServiceNow Mountain View, CA

Mountain View, CA

1mo ago Closed

$140k - $217k/yr Full time Agentic Ai AI Ai Ml

This position is no longer accepting applications (closed Sep 11, 2026).

Job Description

ServiceNow is seeking Machine Learning Engineers for the GenAI Search Platform team to improve search-based question answering using traditional ML and LLM-enabled approaches.

Responsibilities

Lead development of an enterprise search platform with LLM-enabled generative AI features in a fast-paced environment
Enable product teams to prototype and ship new features for customers at scale
Evaluate and iterate on ranking improvements using a systematic approach
Enforce quality, security, privacy, and scalability guarantees for search and summarization
Build systematic evaluation metrics and methodology for search models
Use LLMs to tackle enterprise evaluation challenges, including security and privacy concerns
Collaborate across ranking, product, design, infrastructure, and data science teams to advance agentic search capabilities
Partner with product, UX, customer success, and engineering teams to identify and deliver well-structured solutions

Requirements

3+ years of engineering experience
First-hand experience with search platforms such as:
- traditional information retrieval
- search ranking and relevance
- recommendation systems
- question answering systems
Background in one or more areas: text processing, information retrieval, natural language understanding, or machine learning
Interest in helping product and ranking teams move faster and scale solutions with strong focus on customer impact and success

Technologies

LLMs (large language models)
Natural language understanding
Retrieval augmented generation (RAG)

Location

Mountain View, CA (onsite)

Salary

USD 139,700 - 216,500 per year

Benefits

Health plans, including flexible spending accounts
401(k) plan with company match
ESPP
Matching donations
Flexible time away plan
Family leave programs
Base pay of $139,700 - $216,500, plus equity (when applicable), variable/incentive compensation and benefits

Nice to Have

Experience with, or interest in learning, large language models, RAG, and agents
Experience with Python, including contributing to mature Python libraries or platform code, such as open-source repositories
Experience with Golang (not required)

Work Persona Information

Work personas (flexible, remote, or required in office) are assigned based on the nature of the work and assigned work location
To determine eligibility, ServiceNow may confirm the distance between your primary residence and the closest ServiceNow office using a third-party service

View full job page →

1mo ago

Lead AI Engineer

Capital One New York, NY

New York, NY

1mo ago Closed

$215k - $246k/yr Full time Ai Governance Ai Ml Ai Observability

This position is no longer accepting applications (closed Sep 4, 2026).

Job Description

Lead AI Engineer at Capital One (New York, NY) focuses on responsible, reliable AI systems across training, inference, evaluation, governance, and observability.

Responsibilities

Collaborate with cross-functional partners including engineers, research scientists, technical program managers, and product managers to deliver AI-powered products for associates and customers.
Design, develop, test, deploy, and support AI software components such as foundation model training, large language model inference, similarity search, guardrails, model evaluation, experimentation, governance, and observability.
Apply Open Source and SaaS AI technologies including AWS Ultraclusters, Huggingface, VectorDBs, Nemo Guardrails, PyTorch, and related tooling.
Develop and introduce state-of-the-art LLM optimization methods to improve production performance including scalability, cost, latency, and throughput.
Help shape the technical vision and long-term roadmap for foundational AI systems at Capital One.

Requirements

Bachelor’s or Master’s degree in Computer Science, AI, Electrical Engineering, Computer Engineering, or a related field with the following experience:
- Bachelor’s + at least 4 years developing AI and ML algorithms or technologies, or
- Master’s + at least 2 years developing AI and ML algorithms or technologies.
At least 4 years programming with Python, Go, Scala, or Java.

Preferred Qualifications

6 years deploying scalable and responsible AI solutions on cloud platforms such as AWS, Google Cloud, Azure, or equivalent private cloud.
Experience designing, developing, delivering, and supporting AI services.
Experience developing AI/ML algorithms or technologies (e.g., LLM inference, similarity search and VectorDBs, guardrails, memory) using Python, C++, C#, Java, or Golang.
Experience applying state-of-the-art training and inference optimization techniques to improve hardware utilization, latency, throughput, and cost.
Strong interest in staying current with AI research and applying novel techniques judiciously in production.

Core Technologies

AWS Ultraclusters, Huggingface, VectorDBs, Nemo Guardrails, PyTorch
Foundation model training, large language model inference, similarity search, guardrails
Model evaluation, experimentation, governance, observability
Python, Go, Scala, Java, AWS, Google Cloud, Azure, C++, C#, Golang

Team

The Intelligent Foundations and Experiences (IFX) team brings Capital One’s AI vision to life.
IFX partners across the company to advance science and AI engineering and build and deploy proprietary solutions central to business value for millions of customers.
AI models and platforms help teams enhance products with responsible, scalable AI for high-leverage impact.

Benefits

Comprehensive, competitive, and inclusive health, financial, and other benefits supporting total well-being.
Performance-based incentive compensation, which may include cash bonus(es) and/or long-term incentives (LTI).

Additional Information

Capital One may sponsor new qualified applicants for employment authorization.
Expected application review window: minimum of 5 business days.
No agencies please.
Equal opportunity employer (EOE, including disability/vet) with non-discrimination in compliance with applicable laws.
Promotes a drug-free workplace.
May consider qualified applicants with a criminal history consistent with applicable laws.
Accommodation available via Capital One Recruiting: 1-800-304-9102 or RecruitingAccommodation@capitalone.com.
Recruiting process questions: Careers@capitalone.com.
Capital One does not provide, endorse, or guarantee third-party products, services, educational tools, or other information available through this site.
Capital One Financial includes multiple entities; postings for Canada, the United Kingdom, and the Philippines correspond to specific entities.

Salary

New York, NY: USD 215,200 - 245,600 per year
Candidates hired in other locations will be subject to the pay range associated with that location.
Other listed ranges (full-time annual): Cambridge, MA: $197,300 - $225,100; McLean, VA: $197,300 - $225,100; San Francisco, CA: $215,200 - $245,600; San Jose, CA: $215,200 - $245,600.

View full job page →

Llm Evaluation Engineer Jobs

Machine Learning Engineer, Human Centered AI - Evaluations & Insights

Machine Learning Engineer, GAI Search Platform

Lead AI Engineer

Get Llm Evaluation Engineer Jobs alerts