Evaluation Datasets Engineer Jobs

1 job found

Today

Machine Learning Engineer, Evaluation and Quality, Creativity Apps

Apple San Diego, CA

San Diego, CA

Today

$142k - $263k/yr Full time Ai Testing Artificial Intelligence Evaluation Datasets

Apply Now

Job Description

Apple is seeking a Machine Learning Engineer focused on evaluation and quality for its Creativity Apps. In this onsite role in San Diego, you will shape how AI features are tested, measured, and improved, connecting model development to the real creator experience. You will own evaluation strategy across ML models and LLM-based capabilities, build the datasets and infrastructure that power quality work, and drive findings through triage to resolution.

Role summary

You will define evaluation methodology and metrics for ML models, LLMs, and AI features, curate evaluation datasets, and maintain evaluation infrastructure with reproducible runs and tracked baselines. The work spans end-to-end quality, from model checkpoints to integrated features that ship to customers, with clear reporting that supports both technical and non-technical decisions.

Responsibilities

Own evaluation and quality for AI features across Apple Creativity Apps.
Collaborate with Machine Learning engineering, infrastructure, product, and design teams to translate models and use cases into evaluation methodology, testing strategy, and quality metrics aligned to creator outcomes.
Define evaluation strategy and metrics for ML models, LLMs, and AI features tailored to model type, product context, and use cases.
Curate and create evaluation datasets, measure and improve dataset quality, and run iterative improvement loops to keep datasets representative.
Design, build, maintain, and optimize evaluation infrastructure at scale, including reproducible runs, tracked baselines, and version-to-version comparisons.
Apply ML, LLMs, and agentic workflows to evaluation, including automated judging, test generation, and analysis to improve efficiency, accuracy, and scalability.
Drive end-to-end evaluation and quality from model checkpoints through shipping features, including triaging issues and driving them to resolution.
Report quality clearly by converting evaluation results into ship/no-ship risk and recommendations for technical and non-technical audiences.
Partner with cross-functional teams, working effectively in ambiguous situations and making informed, critical judgments.

Requirements

5+ years of industry experience in machine learning or software engineering, with a focus on ML/AI model evaluation and quality.
Experience with evaluation methodology for ML models, LLMs, and AI products or features.
Strong understanding of the ML development cycle, including data and training, evaluation, deployment, and monitoring.
Solid ML fundamentals and LLM knowledge, including agentic workflows, with the ability to keep up with field developments.
Track record applying LLMs and/or agentic workflows to ML evaluation and quality.
Experience with data curation, dataset creation, and dataset quality, including iterative improvement workflows for product-specific datasets.
Experience designing, implementing, maintaining, or optimizing ML evaluation infrastructure at scale.
Experience designing evaluation metrics based on the model, product, and use cases.
Experience driving end-to-end evaluation and quality for AI products.
Fluency in quality reporting, including presenting quality data to technical and non-technical audiences.
Strong communication skills and experience working with and driving cross-functional partners.
Proficiency in Python and/or Swift or comparable languages.
Bachelor’s degree in Computer Science, Data Science, Machine Learning, or a related field, or equivalent experience.

Technologies

Python
Swift
iOS
macOS

Benefits

Comprehensive medical and dental coverage
Retirement benefits
A range of discounted products and free services
Reimbursement for certain educational expenses (including tuition) for formal education related to advancing your career
Opportunity to become an Apple shareholder through participation in Apple’s discretionary employee stock programs
Discretionary restricted stock unit awards
Ability to purchase Apple stock at a discount if voluntarily participating in the Employee Stock Purchase Plan
Discretionary bonuses or commission payments as well as relocation (might be eligible)
Base pay range between $142,300 and $263,300 (determined within a range based on skills, qualifications, experience, and location)

Preferred qualifications

Knowledge of the software development life cycle and software quality assurance methodologies.
Experience in objective image and video quality evaluation and metrics.
Knowledge of color science and/or experience with subjective image or video quality review.
Passion for creative applications; experience as a creator (photo, video, music, or design) is a plus.
Excitement about learning new technology and applying it to practical problems.
Familiarity with iOS and macOS.

View full job page →

Get Evaluation Datasets Engineer Jobs alerts

New Evaluation Datasets Engineer Jobs, daily.