Machine Learning Evaluation

Machine Learning Evaluation Engineer Jobs

6 jobs found

Today

Evaluation & Insights Machine Learning Engineer

Apple Cupertino, CA

Cupertino, CA

Today

$185k - $325k/yr Full time Ai Evaluation Ai Testing And Evaluation Artificial Intelligence

Apply Now

Job Description

On Apple’s Human-Centered AI team, this role focuses on evaluating and improving AI systems using data-driven analysis, model behavior study, and qualitative insights that translate into engineering action.

Location

Cupertino, CA (onsite)

Compensation

USD 184,700 - 324,800 per year. Base pay depends on skills, qualifications, experience, and location.

Role Summary

The Evaluation & Insights Machine Learning Engineer designs evaluation approaches to assess LLMs and multimodal systems. The engineer builds evaluation frameworks, analyzes AI outputs, identifies failure patterns, and communicates findings to product and engineering teams to support model improvements.

Key Responsibilities

Architect and deliver comprehensive evaluation suites for LLMs and multimodal models, with focus on edge cases across multi-step reasoning, factuality, adversarial robustness, safety, and alignment.
Create deterministic, heuristic, and LLM-assisted evaluation frameworks (including LLM-as-a-judge and reward modeling) to measure human-perceived quality metrics such as helpfulness and hallucination rates.
Convert qualitative failure modes into quantifiable loss patterns, programmatic guardrails, and actionable data-mixture adjustments for model training and inference.
Work with engineering teams to refine model behavior using evaluation telemetry to guide prompt engineering, Retrieval-Augmented Generation (RAG) strategies, and model fine-tuning.
Use advanced ML methods such as embedding-based clustering, representation learning, and perturbation analysis to map error taxonomies and latent failure manifolds.
Build MLOps workflows that codify evaluation metrics, automate regression testing across model checkpoints, and integrate human-centric assessments into ML CI/CD pipelines.
Design scalable, distributed inference and processing pipelines (for example, Ray and vLLM) to support high-throughput evaluation, automated annotation, and large-scale output analysis.
Define quantitative evaluation frameworks covering nuanced human factors, including trust calibration, conversational state tracking, and interpretability.
Develop automated evaluation pipelines using LLMs to score outputs at scale, with optimization for strong correlation to human baseline annotations.
Collaborate with ML researchers, software developers, and product managers to turn product requirements into reliable and efficient evaluation infrastructure.

Required Qualifications

Knowledge of human factors, HCI, or cognitive science methodologies as applied to AI system design.
Bachelor’s or Master’s degree in Computer Science, Machine Learning, Artificial Intelligence, Cognitive Science, or a related technical field.
8+ years of relevant industry experience in ML Engineering or Applied Research.
Advanced proficiency in Python and modern deep learning ecosystems (PyTorch, JAX, Hugging Face).
Proven experience building scalable ML inference pipelines, model-evaluation workflows, and structured rating frameworks for large-scale AI systems.
Ability to interpret unstructured model outputs (text, transcripts, embedding spaces) and synthesize qualitative findings into actionable engineering guidance and training objectives.
Hands-on experience developing, fine-tuning, or evaluating LLMs, multimodal models, and NLP systems.
Deep familiarity with AI quality metrics, hallucination detection techniques (for example, SelfCheckGPT), model alignment approaches (such as RLHF and DPO), and LLM-as-a-judge frameworks (for example, G-Eval and DeepEval).
Experience building internal tools or automated pipelines for ML workflows using platforms such as MLflow and Weights & Biases (or similar).
Strong familiarity with advanced prompt engineering, RAG architectures (vector databases and semantic search), and Fine-Tuning.

Preferred Qualifications

Knowledge of human factors, HCI, or cognitive science methodologies as applied to AI system design.

Technologies & Tools

Python, PyTorch, JAX, Hugging Face
LLM-as-a-judge, reward modeling
Retrieval-Augmented Generation (RAG), vector databases, semantic search
Embedding-based clustering, representation learning, perturbation analysis
MLOps, CI/CD pipelines
Ray, vLLM
Trust calibration
SelfCheckGPT, RLHF, DPO
G-Eval, DeepEval
MLflow, Weights & Biases
Fine-Tuning

Benefits

Comprehensive medical and dental coverage
Retirement benefits
A range of discounted products and free services
Reimbursement for certain educational expenses, including tuition
Discretionary restricted stock unit awards
Opportunity to purchase Apple stock at a discount through the Employee Stock Purchase Plan
Opportunity to progress as you grow and develop within a role
Comprehensive total compensation package may include discretionary bonuses or commission payments as well as relocation

Additional Notes on Pay and Programs

Apple benefit, compensation, and employee stock programs are subject to eligibility requirements and other terms of the applicable plan or program. This role might be eligible for discretionary bonuses or commission payments and relocation.

View full job page →

Today

Machine Learning Engineer

XPO Boston, MA

Boston, MA

Today

$100k - $120k/yr Full time Artificial Intelligence Automation Bigquery

Apply Now

Job Description

XPO is hiring a Machine Learning Engineer to help build and sustain the machine learning foundation behind training, evaluation, and deployment. This hybrid role is based in Boston, MA, with a focus on reliable data preparation and validation, ML infrastructure, and MLOps capabilities that support production performance over time.

The position includes hands-on work across the ML lifecycle, from building pipelines that ensure model-ready data to implementing CI/CD workflows for ML models. You will also help connect experimentation to deployment by partnering with applied and data scientists and collaborating closely with data engineering teams.

Role summary

Build and maintain ML data preparation and validation tooling to support high-quality inputs for ML and optimization models
Design and implement ML infrastructure for model training, evaluation, and deployment
Strengthen MLOps with CI/CD, monitoring, and feedback loops to track production performance
Contribute to shared MLOps tooling and best practices across the AI/ML organization

Responsibilities

Develop and maintain data preparation and validation tooling to improve the quality of ML and optimization model inputs
Create ML infrastructure that supports training, evaluation, and deployment workflows
Build and maintain CI/CD pipelines for machine learning models, including automated testing and validation
Implement model monitoring, drift detection, and feedback loops to measure and improve production performance
Partner with applied and data scientists to productionize models and streamline movement from experimentation to deployment
Work with data engineering teams to ensure data pipelines are reliable and accessible
Support shared MLOps tooling and organizational best practices

What you’ll do on a typical day

Build and maintain data preparation and validation tooling
Design and implement ML infrastructure for training, evaluation, and deployment
Maintain CI/CD pipelines for machine learning models with automated testing and validation
Implement monitoring, drift detection, and feedback loops for production models
Collaborate with applied and data scientists to productionize models
Coordinate with data engineering teams to ensure dependable, accessible data pipelines
Contribute to shared MLOps tooling and best practices

Requirements

Bachelor's degree in Computer Science, Engineering, or related field, or equivalent related work or military experience
1 year of experience in software or machine learning engineering, including hands-on experience building data pipelines, ML infrastructure, or MLOps tooling
Experience developing data preparation, validation, or quality-checking tooling for ML pipelines
Proficiency in Python and SQL
Experience with cloud data or ML platforms (examples: AWS, GCP, BigQuery)
Strong collaboration skills partnering with data science/applied science teams and data engineering teams
Master's degree in Computer Science or related field
3+ years of experience building ML infrastructure for training, evaluation, and deployment at scale
Experience building and maintaining CI/CD pipelines for ML models
Experience with model serving and inference infrastructure (batch and real-time)
Experience implementing model monitoring, drift detection, and feedback-loop tooling
Experience with containerization and orchestration (Docker, Kubernetes)
Experience partnering with data engineering teams on data pipeline reliability and access

Technologies

Python
SQL
AWS
GCP
BigQuery
Docker
Kubernetes

Compensation and location

Location: Boston, MA (hybrid)
Annual salary range: USD $100,000 - $120,000 per year
Actual compensation may vary based on experience and skill set
This is an incentive-based position, which may include bonuses, incentive or commission plans

Benefits

Competitive compensation package
Full health insurance benefits available on day one
Life and disability insurance
Earn up to 15 days of PTO over your first year
9 paid company holidays
401(k) option with company match
Education assistance
Opportunity to participate in a company incentive plan

View full job page →

1d ago

Machine Learning Engineer

Johns Hopkins Applied Physics Laboratory (APL) Laurel, MD

Laurel, MD

1d ago

$100k - $245k/yr Full time Ai Engineer Ai Ml Artificial Intelligence

Apply Now

Job Description

Machine Learning Engineer role at Johns Hopkins Applied Physics Laboratory (APL) focused on applying modern AI to national defense non-kinetic systems in an onsite setting in Laurel, MD.

Responsibilities

Design, implement, and evaluate advanced machine learning algorithms for planning, perception, coordination, and control challenges
Build software pipelines that connect data streams, simulation environments, and intelligent decision-making components
Apply cutting-edge AI methods including deep reinforcement learning, foundation models, and large language models
Develop and work with neural network architectures such as convolutional, recurrent, and graph neural networks
Apply computer vision techniques in support of system objectives
Use physics-based modeling and simulation tools to support algorithm development and analysis
Collaborate with scientists and engineers within the group and across APL
Engage with sponsors to communicate proposed concepts, solutions, and supporting analysis

Requirements

Bachelor’s degree in Mathematics, Physics, Engineering, Computer Science, or a related field
At least 2+ years of experience in machine learning and data science
At least 1 year of hands-on experience applying or developing ML algorithms using common libraries such as PyTorch or TensorFlow
Strong foundational knowledge in at least two of the following areas: classification, clustering, deep learning, reinforcement learning, computer vision (object detection and visual tracking), multi-agent systems, optimization/control theory
Demonstrated experience working with version control software such as Git
Strong verbal and written communication skills
Ability to obtain an Interim Secret security clearance by the start date and ultimately obtain a Secret clearance; U.S. citizenship required

Technologies

PyTorch, TensorFlow, Git
Deep reinforcement learning
Foundation models, large language models
Convolutional neural networks, recurrent neural networks, graph neural networks
Computer vision
Physics-based modeling and simulation tools

Minimum and Maximum Salary

Minimum: USD 100,000 per year
Maximum: USD 245,000 per year

Preferred Qualifications

MS in Mathematics, Physics, Engineering, Computer Science, or a related field
5+ years of experience designing and implementing AI/ML algorithms across a variety of datasets
Proven experience applying state-of-the-art deep learning techniques to solve distributed resource allocation problems
Hands-on experience building computer vision pipelines for detection, tracking, segmentation, or multi-modal sensor fusion
Experience with modeling and simulation platforms such as AFSIM, Blender, Unity, or Unreal
Comfort working in high performance computing environments (GPU/CPU clusters)
Proficiency in one or more technology areas: multi-agent reinforcement learning, geometric deep learning, multi-modal sensor fusion, agentic AI
Track record of writing deployable, production-level code in Python and C/C++ for real-world applications

View full job page →

4d ago

Senior Machine Learning Engineer

Motion Recruitment Boston, MA

Boston, MA

4d ago

Full time Senior Artificial Intelligence Automation

Apply Now

Job Description

This Senior Machine Learning Engineer role is focused on end-to-end ownership of machine learning for medical-device and robotics or automation applications, from sensor-driven data workflows through model deployment to constrained hardware.

Onsite Location

Boston, MA 02116 (onsite)

Role Summary

The position covers the full machine learning lifecycle for real-time device use cases. You will build and deploy ML models that translate sensor data into reliable outputs on constrained platforms. Collaboration with embedded, firmware, and software teams is a core part of the work, alongside contributions to MLOps and validation, including regulatory-ready documentation.

Responsibilities

Develop and troubleshoot workflows for collecting, cleaning, and organizing sensor data.
Build and refine ML models aimed at real-time device applications and performance improvements.
Partner with firmware teams to embed and test AI features on hardware platforms.
Set up and oversee tools for experiment tracking, automated evaluations, and deployment management.
Analyze model behavior, maintain reliability, and resolve issues to support high-quality outputs.

Technical Skills

Key technologies involved include Python, PyTorch, TensorFlow, and deployment toolchains such as TFLite, ONNX, CoreML, and TensorRT (or equivalent). The role also expects proficiency with MLOps, along with software engineering fundamentals supported by C and C++.

Requirements

Strong proficiency in Python.
Hands-on experience with PyTorch or TensorFlow.
Experience deploying models to edge using TFLite, ONNX, CoreML, TensorRT, or equivalent.
Experience building sensor data pipelines.
Proficiency with MLOps.
Solid software engineering fundamentals.
Proficiency in C or C++.

Desired Skills and Experience

Experience with physiological signal processing for medical or wearable applications.
Background in robotics or autonomous systems.
Experience working in a startup or small team.
Degree in a relevant field.

Work Experience Level

Minimum experience: 5 years

Daily Responsibilities

100% Hands On

Benefits

Bonus OR Commission eligible
Medical Insurance
Dental Benefits
Vision Benefits
Paid Time Off (PTO)
401(k) (including match, if applicable)

View full job page →

5d ago

Machine Learning Engineer

UST Portland, OR

Portland, OR

5d ago

$82k - $123k/yr Full time Agent Orchestration Agentic Ai Agentic Ai Orchestration

Apply Now

Job Description

UST is seeking a Machine Learning Engineer (ML Engineer II) to build and optimize multi-agent systems for real-world performance. This onsite role in Oregon focuses on taking agent workloads from application design through inference runtime behavior and down to accelerator hardware and middleware layers.

What you’ll do

Develop a multi-agent workload at the application level, then profile and optimize its execution all the way through the inference runtime, OMIX/middleware, compute runtime, Linux GPU driver, and accelerator hardware.
Improve key performance metrics including latency, throughput, tokens/sec, memory footprint, and accelerator utilization.
Select and tune models and inference engines based on agent workload characteristics, hardware capabilities, and performance requirements.

What you bring

Hands-on development experience with multi-agent workloads.
Experience with agent frameworks such as LangGraph/LangChain, AutoGen, and CrewAI.
Strong understanding of LLMs, SLMs, and multimodal models, including Transformer architecture, attention, tokenization, context windows, and KV cache concepts.
Hands-on experience selecting, evaluating, and deploying models for different agentic workload requirements.
Knowledge of model formats and optimization workflows, including ONNX, OpenVINO IR, safe tensors, and quantization (FP16, BF16, INT8, INT4).
Experience with inference engines and runtimes such as OpenVINO, ONNX Runtime, vLLM, llama.cpp, or TGI.
Understanding of prefill vs. decode, batching, continuous batching, speculative decoding, KV-cache management, and memory optimization.
Solid grasp of CPU/GPU execution, device placement, and heterogeneous inference.
Ability to benchmark and compare models, inference engines, and runtime configurations, then identify performance bottlenecks.
Knowledge of GPU memory, kernel execution, synchronization, device selection, and host/device data movement.
Familiarity with middleware and accelerator compute runtimes such as OMIX/OneAPI/SYCL, Level Zero, or OpenCL.
Python strength, plus working knowledge of C/C++.
Experience with Git/GitHub, Linux shell, and debugging tools.
Practical use of GitHub Copilot for development, debugging, and code generation.
Docker and container fundamentals.
Ability to benchmark and profile AI workloads.
Understanding of latency, throughput, tokens/sec, GPU utilization, memory bandwidth, and CPU/GPU bottlenecks, including identifying whether a performance issue comes from the agent, model, inference engine, runtime, or driver.

Core technologies

LangGraph, LangChain, AutoGen, CrewAI
Python, C/C++, Git, GitHub, GitHub Copilot
Docker
ONNX, Open VINO IR, safe tensors
FP16, BF16, INT8, INT4
Open VINO, ONNX Runtime, vLLM, llama.cpp, TGI
OMIX, OneAPI, SYCL, Level Zero, OpenCL

Role location

Oregon (onsite)

Compensation

$82,000 - $123,000 USD per year

Benefits

Full-time employees accrue a minimum of 10 days of paid vacation per year.
6 days of paid sick leave each year (pro-rated for new hires throughout the year).
10 paid holidays.
Eligible for paid bereavement leave and jury duty.
401(k) Retirement Plan with employer matching.
Medical, dental, and vision insurance for employees and dependents residing in the US.
Company-paid Employee Only benefits: basic life insurance, accidental death and disability insurance, and short- and long-term disability benefits.
Regular employees may purchase additional voluntary short-term disability benefits.
Eligible to participate in a Health Savings Account (HSA).
Eligible to participate in a Flexible Spending Account (FSA) for healthcare, dependent child care, and/or commuting expenses.

Additional skills

Agentic AI
Python
Linux
MCP3
RAG
Kubernetes

View full job page →

5d ago

AI Engineer - Algorithm Evaluation & Agentic Systems

Apple Sunnyvale, CA

Sunnyvale, CA

5d ago

$150k - $278k/yr Full time Agentic Ai Agentic Ai Orchestration Ai Agent

Apply Now

Job Description

On Apple’s DAQ team, you will focus on evaluating and improving advanced visual technologies. In this role, you will work at the intersection of algorithm evaluation and agentic systems, helping translate model behavior into measurable performance and production-ready workflows for advanced computer vision and video understanding.

Working within DAQ, your contribution centers on building evaluation pipelines, diagnosing failures, and architecting autonomous multi-modal solutions that connect experimentation with real-world deployment. You will lead benchmarking and integration of state-of-the-art image and video understanding models, while partnering with core model training teams to deliver data-driven insights for iteration and fine-tuning.

Responsibilities

Design, build, and scale comprehensive evaluation pipelines for both end-to-end system evaluation and granular component-level testing on complex image and video understanding tasks.
Analyze model outputs to identify root causes of visual hallucinations, temporal inconsistencies in video, and edge-case failures.
Build, deploy, and evaluate agentic workflows that use vision models to autonomously solve multi-step user problems such as video summarization and visual search.
Use component-level evaluation to isolate which parts of an agentic workflow, including tool selection, memory retrieval, and visual reasoning, succeed or fail.
Lead strategy for curating high-quality schematized datasets and ground-truth benchmarks tailored for evaluating multi-modal capabilities.
Partner with core model training teams to provide actionable metrics and insights that guide subsequent model training and fine-tuning.
Lead benchmarking and integration of state-of-the-art models for image and video understanding.

Requirements

MS and a minimum of 3 years relevant industry experience.
3+ years of applied experience in Machine Learning, Computer Vision, or AI System Evaluation.
Deep understanding of core machine learning principles, including probability, statistics, data distributions, and model bias/variance.
Deep theoretical and practical understanding of computer vision and vision-language models, including how Vision Transformers (ViTs), spatial-temporal modeling, and image/video processing work.
Proven ability to define robust metrics and KPIs and design rigorous evaluation frameworks for generative AI or foundation models, including custom benchmark creation, automated regression testing, LLM/VLM-as-a-judge methodologies, and human-in-the-loop evaluation.
Experience building and evaluating LLM/VLM-powered agents, including tool use, multi-step reasoning, planning, and memory management workflows.
Strong ability to probe ML models for edge cases, hallucinations, and performance bottlenecks in constrained environments, and translate findings into improvement recommendations.
Strong proficiency in Python and experience with PyTorch, including running inference, extracting embeddings, and building scalable evaluation pipelines.

Technologies

Python
PyTorch
Vision Transformers (ViTs)
Computer Vision (CV)
Vision-Language Models (VLMs)
LLM/VLM-as-a-judge methodologies
LLM/VLM-powered agents

Benefits

Comprehensive medical and dental coverage
Retirement benefits
Discounted products and free services
Reimbursement for certain educational expenses, including tuition for formal education related to advancing your career
Opportunity to become an Apple shareholder through participation in Apple’s discretionary employee stock programs
Discretionary restricted stock unit awards
Ability to purchase Apple stock at a discount through participation in the Employee Stock Purchase Plan
Potential discretionary bonuses or commission payments
Relocation (may be eligible)

What We Value

Production mindset: correctness, observability, and maintainability
Ability to reason about system-level tradeoffs, not only model performance
Ability to balance experimentation speed with engineering rigor
Comfort working in ambiguous spaces and defining metrics from first principles
Clear communication of technical findings to both technical and non-technical audiences

Preferred Qualifications

Demonstrated ability to lead technical evaluation strategies end-to-end, drive architectural decisions for testing infrastructure, and mentor engineers.
Strong foundation in statistics, including hypothesis testing, confidence intervals, and experimental design.
Knowledge of reinforcement learning, planning, or decision-making systems.
Experience evaluating multi-modal or multi-agent systems.
Prior work on AI reliability, safety, or benchmarking.

Pay & Benefits

Base pay range: $150,400 to $277,600 per year
Employees may have opportunities to become shareholders through discretionary employee stock programs
Eligible for discretionary restricted stock unit awards
Ability to purchase Apple stock at a discount if participating in the Employee Stock Purchase Plan
Role might be eligible for discretionary bonuses or commission payments and relocation
Learn more about Apple Benefits
Note: Benefit, compensation, and employee stock program eligibility are subject to requirements and terms of the applicable plan or program.

Location: Sunnyvale, CA (onsite)

Minimum experience: 3 years

Education: MS

Compensation: USD 150,400 - 277,600 per yearly

View full job page →

Machine Learning Evaluation Engineer Jobs

Evaluation & Insights Machine Learning Engineer

Machine Learning Engineer

Machine Learning Engineer

Senior Machine Learning Engineer

Machine Learning Engineer

AI Engineer - Algorithm Evaluation & Agentic Systems

Get Machine Learning Evaluation Engineer Jobs alerts