Llm As Judge Engineer Jobs

1 job found

Today

AI Engineer - Algorithm Evaluation & Agentic Systems

Apple Sunnyvale, CA

Sunnyvale, CA

Today

$150k - $278k/yr Full time Agentic Ai Agentic Ai Orchestration Ai Agent

Apply Now

Job Description

On Apple’s DAQ team, you will focus on evaluating and improving advanced visual technologies. In this role, you will work at the intersection of algorithm evaluation and agentic systems, helping translate model behavior into measurable performance and production-ready workflows for advanced computer vision and video understanding.

Working within DAQ, your contribution centers on building evaluation pipelines, diagnosing failures, and architecting autonomous multi-modal solutions that connect experimentation with real-world deployment. You will lead benchmarking and integration of state-of-the-art image and video understanding models, while partnering with core model training teams to deliver data-driven insights for iteration and fine-tuning.

Responsibilities

Design, build, and scale comprehensive evaluation pipelines for both end-to-end system evaluation and granular component-level testing on complex image and video understanding tasks.
Analyze model outputs to identify root causes of visual hallucinations, temporal inconsistencies in video, and edge-case failures.
Build, deploy, and evaluate agentic workflows that use vision models to autonomously solve multi-step user problems such as video summarization and visual search.
Use component-level evaluation to isolate which parts of an agentic workflow, including tool selection, memory retrieval, and visual reasoning, succeed or fail.
Lead strategy for curating high-quality schematized datasets and ground-truth benchmarks tailored for evaluating multi-modal capabilities.
Partner with core model training teams to provide actionable metrics and insights that guide subsequent model training and fine-tuning.
Lead benchmarking and integration of state-of-the-art models for image and video understanding.

Requirements

MS and a minimum of 3 years relevant industry experience.
3+ years of applied experience in Machine Learning, Computer Vision, or AI System Evaluation.
Deep understanding of core machine learning principles, including probability, statistics, data distributions, and model bias/variance.
Deep theoretical and practical understanding of computer vision and vision-language models, including how Vision Transformers (ViTs), spatial-temporal modeling, and image/video processing work.
Proven ability to define robust metrics and KPIs and design rigorous evaluation frameworks for generative AI or foundation models, including custom benchmark creation, automated regression testing, LLM/VLM-as-a-judge methodologies, and human-in-the-loop evaluation.
Experience building and evaluating LLM/VLM-powered agents, including tool use, multi-step reasoning, planning, and memory management workflows.
Strong ability to probe ML models for edge cases, hallucinations, and performance bottlenecks in constrained environments, and translate findings into improvement recommendations.
Strong proficiency in Python and experience with PyTorch, including running inference, extracting embeddings, and building scalable evaluation pipelines.

Technologies

Python
PyTorch
Vision Transformers (ViTs)
Computer Vision (CV)
Vision-Language Models (VLMs)
LLM/VLM-as-a-judge methodologies
LLM/VLM-powered agents

Benefits

Comprehensive medical and dental coverage
Retirement benefits
Discounted products and free services
Reimbursement for certain educational expenses, including tuition for formal education related to advancing your career
Opportunity to become an Apple shareholder through participation in Apple’s discretionary employee stock programs
Discretionary restricted stock unit awards
Ability to purchase Apple stock at a discount through participation in the Employee Stock Purchase Plan
Potential discretionary bonuses or commission payments
Relocation (may be eligible)

What We Value

Production mindset: correctness, observability, and maintainability
Ability to reason about system-level tradeoffs, not only model performance
Ability to balance experimentation speed with engineering rigor
Comfort working in ambiguous spaces and defining metrics from first principles
Clear communication of technical findings to both technical and non-technical audiences

Preferred Qualifications

Demonstrated ability to lead technical evaluation strategies end-to-end, drive architectural decisions for testing infrastructure, and mentor engineers.
Strong foundation in statistics, including hypothesis testing, confidence intervals, and experimental design.
Knowledge of reinforcement learning, planning, or decision-making systems.
Experience evaluating multi-modal or multi-agent systems.
Prior work on AI reliability, safety, or benchmarking.

Pay & Benefits

Base pay range: $150,400 to $277,600 per year
Employees may have opportunities to become shareholders through discretionary employee stock programs
Eligible for discretionary restricted stock unit awards
Ability to purchase Apple stock at a discount if participating in the Employee Stock Purchase Plan
Role might be eligible for discretionary bonuses or commission payments and relocation
Learn more about Apple Benefits
Note: Benefit, compensation, and employee stock program eligibility are subject to requirements and terms of the applicable plan or program.

Location: Sunnyvale, CA (onsite)

Minimum experience: 3 years

Education: MS

Compensation: USD 150,400 - 277,600 per yearly

View full job page →

Get Llm As Judge Engineer Jobs alerts

New Llm As Judge Engineer Jobs, daily.