On Apple’s Human-Centered AI team, this role focuses on evaluating and improving AI systems using data-driven analysis, model behavior study, and qualitative insights that translate into engineering action.
Location
Cupertino, CA (onsite)
Compensation
USD 184,700 - 324,800 per year. Base pay depends on skills, qualifications, experience, and location.
Role Summary
The Evaluation & Insights Machine Learning Engineer designs evaluation approaches to assess LLMs and multimodal systems. The engineer builds evaluation frameworks, analyzes AI outputs, identifies failure patterns, and communicates findings to product and engineering teams to support model improvements.
Key Responsibilities
- Architect and deliver comprehensive evaluation suites for LLMs and multimodal models, with focus on edge cases across multi-step reasoning, factuality, adversarial robustness, safety, and alignment.
- Create deterministic, heuristic, and LLM-assisted evaluation frameworks (including LLM-as-a-judge and reward modeling) to measure human-perceived quality metrics such as helpfulness and hallucination rates.
- Convert qualitative failure modes into quantifiable loss patterns, programmatic guardrails, and actionable data-mixture adjustments for model training and inference.
- Work with engineering teams to refine model behavior using evaluation telemetry to guide prompt engineering, Retrieval-Augmented Generation (RAG) strategies, and model fine-tuning.
- Use advanced ML methods such as embedding-based clustering, representation learning, and perturbation analysis to map error taxonomies and latent failure manifolds.
- Build MLOps workflows that codify evaluation metrics, automate regression testing across model checkpoints, and integrate human-centric assessments into ML CI/CD pipelines.
- Design scalable, distributed inference and processing pipelines (for example, Ray and vLLM) to support high-throughput evaluation, automated annotation, and large-scale output analysis.
- Define quantitative evaluation frameworks covering nuanced human factors, including trust calibration, conversational state tracking, and interpretability.
- Develop automated evaluation pipelines using LLMs to score outputs at scale, with optimization for strong correlation to human baseline annotations.
- Collaborate with ML researchers, software developers, and product managers to turn product requirements into reliable and efficient evaluation infrastructure.
Required Qualifications
- Knowledge of human factors, HCI, or cognitive science methodologies as applied to AI system design.
- Bachelor’s or Master’s degree in Computer Science, Machine Learning, Artificial Intelligence, Cognitive Science, or a related technical field.
- 8+ years of relevant industry experience in ML Engineering or Applied Research.
- Advanced proficiency in Python and modern deep learning ecosystems (PyTorch, JAX, Hugging Face).
- Proven experience building scalable ML inference pipelines, model-evaluation workflows, and structured rating frameworks for large-scale AI systems.
- Ability to interpret unstructured model outputs (text, transcripts, embedding spaces) and synthesize qualitative findings into actionable engineering guidance and training objectives.
- Hands-on experience developing, fine-tuning, or evaluating LLMs, multimodal models, and NLP systems.
- Deep familiarity with AI quality metrics, hallucination detection techniques (for example, SelfCheckGPT), model alignment approaches (such as RLHF and DPO), and LLM-as-a-judge frameworks (for example, G-Eval and DeepEval).
- Experience building internal tools or automated pipelines for ML workflows using platforms such as MLflow and Weights & Biases (or similar).
- Strong familiarity with advanced prompt engineering, RAG architectures (vector databases and semantic search), and Fine-Tuning.
Preferred Qualifications
- Knowledge of human factors, HCI, or cognitive science methodologies as applied to AI system design.
Technologies & Tools
- Python, PyTorch, JAX, Hugging Face
- LLM-as-a-judge, reward modeling
- Retrieval-Augmented Generation (RAG), vector databases, semantic search
- Embedding-based clustering, representation learning, perturbation analysis
- MLOps, CI/CD pipelines
- Ray, vLLM
- Trust calibration
- SelfCheckGPT, RLHF, DPO
- G-Eval, DeepEval
- MLflow, Weights & Biases
- Fine-Tuning
Benefits
- Comprehensive medical and dental coverage
- Retirement benefits
- A range of discounted products and free services
- Reimbursement for certain educational expenses, including tuition
- Discretionary restricted stock unit awards
- Opportunity to purchase Apple stock at a discount through the Employee Stock Purchase Plan
- Opportunity to progress as you grow and develop within a role
- Comprehensive total compensation package may include discretionary bonuses or commission payments as well as relocation
Additional Notes on Pay and Programs
Apple benefit, compensation, and employee stock programs are subject to eligibility requirements and other terms of the applicable plan or program. This role might be eligible for discretionary bonuses or commission payments and relocation.
