Machine Learning Engineer, Evaluation

Lead responsible AI evaluation for global generative AI products by building metrics, datasets, and scalable safety evaluation pipelines.

Responsibilities

• Develop evaluation metrics focused on safety and fairness risks in generative models and Gen-AI features
• Design evaluation datasets, define data needs, and scale data coverage using human and synthetic generation approaches
• Create sampling strategies and combine human annotation with auto-grading for high-confidence insights at scale
• Implement data pipelines and collaborate cross-functionally to deliver end-to-end safety evaluations
• Improve evaluation operations by automating manual workflows, increasing evaluation velocity, and scaling infrastructure for rapid release cycles
• Translate evaluation findings into actionable insights for product engineering teams and safety policy development
• Work with highly sensitive content, including exposure to offensive and controversial material
• Provide technical direction to operationalize policy and risk assessments into measurable criteria
• Develop LLM- and human-in-the-loop methods for data analysis and guide failure investigations
• Remain hands-on analyzing evaluation outcomes and using human and automated feedback to improve feature safety and user trust
• Foster inclusive technical excellence, empower senior leadership with safety recommendations, and remove team roadblocks
• Conduct scientific investigation and interpretation, including negotiating requirements and defending safety metrics to leadership; report metrics to stakeholders and decision-makers

Requirements

• Experience working in Responsible AI
• Demonstrated curiosity about fairness and bias in generative AI systems, with a strong interest in making technology more equitable
• Social Science or Humanities background, or other demonstrable interest in socio-technical issues
• Prior scientific research and publication experience
• Proven record contributing to diverse teams in a collaborative environment
• Experience with multimodal models, agentic systems, and/or diverse languages and locales
• Experience addressing compliance and regulatory needs in AI, specifically child safety, clinical/health applications, or companionship chatbots
• Experience conducting large-scale benchmarking across models and capabilities
• MS or PhD in Computer Science, Linguistics, Cognitive Science, HCI, Psychology, Mathematics, Physics, or a similar science or technology field with a strong basis in scientific data collection and analysis, plus at least 4 years of relevant work experience
• Alternative qualification: BA/BS with 8+ years of relevant work experience, including 2+ years directly managing engineering teams
• Experience gathering and analyzing varied language, image, voice, video, and/or multi-modal data, including human- and LLM-generated data
• Experience using LLM-Judges and human-in-the-loop methods, including designing human annotation projects, writing guidelines, and handling highly multi-labeled, nuanced, and often conflicting data
• Proficiency in data science, machine learning, analytics, and programming with Python and Pandas; experience with plotting and visualization libraries
• Cross-functional collaboration skills: prioritize competing work, set and maintain milestone schedules, and communicate with team members and internal/external stakeholders
• A rigorous approach to metric development, interpretation of experiments/evaluations, and presenting results to executives
• Comfort leading a team that works with sensitive or potentially offensive content

Technologies

• Python
• Pandas

Pay & Benefits

• Base pay range: $184,700 - $324,800 per year
• Base pay determined based on skills, qualifications, experience, and location
• Comprehensive medical and dental coverage
• Retirement benefits
• Range of discounted products and free services
• Reimbursement for certain educational expenses, including tuition
• Discretionary bonuses or commission payments, plus relocation (if eligible)
• Apple shareholder opportunity through participation in Apple’s discretionary employee stock programs
• Eligible for discretionary restricted stock unit awards
• Option to purchase Apple stock at a discount through voluntary participation in the Employee Stock Purchase Plan
• Note: Benefits, compensation, and stock programs subject to eligibility requirements and plan or program terms

Location: Cupertino, CA (onsite)

Minimum experience: 4 years

Education: MS or PhD