Ml Experimentation Engineer Jobs

1 job found

Today

Machine Learning Engineer – Evals

Mango Global Technologies New York, NY

New York, NY

Today

$220k - $300k/yr Full time Artificial Intelligence Data Science Machine Learning Engineer

Apply Now

Job Description

Mango Global Technologies is hiring a Machine Learning Engineer – Evals in New York, NY (onsite) to own the end-to-end evaluation of an AI memory and identity system. This role focuses on building production-quality evaluation infrastructure, turning experiment results into actionable fixes, and continuously evolving evaluation definitions as models and product capabilities change.

What you’ll do

Define how to determine whether evolving entity representations are improving.
Build the infrastructure needed to measure representation improvements reliably.
Use evaluation results to identify and fix failures across the evaluation loop.
Define what “better” means for representations that evolve over time.
Design novel evaluation methodologies when traditional ground truth is unavailable or insufficient.
Develop scoring frameworks, metrics, rubrics, judge models, and diagnostic methodologies.
Continuously evolve evaluation definitions as models and product capabilities shift.
Build production-quality Python pipelines and evaluation harnesses.
Manage datasets, labels, experiment versions, model versions, judges, reruns, and regression testing.
Create infrastructure that helps researchers and engineers answer new evaluation questions quickly.
Integrate experiment tracking and observability into evaluation workflows.
Analyze model outputs, agent trajectories, and execution traces to find meaningful failure modes.
Translate evaluation findings into concrete engineering or research improvements.
Run experiments, compare results, document findings, and drive the resulting fixes.
Develop and operate simulation environments that model large numbers of entities.
Increase the throughput and reliability of evaluation runs.
Enable rapid iteration across models, prompts, representations, and evaluation strategies.
Own the full process from research question → evaluation design → data → implementation → execution → analysis → diagnosis → rerun → recommendation.

Required qualifications

3+ years of professional experience designing and owning LLM evaluations.
Demonstrated experience designing, building, and running LLM evaluations end to end.
Production-quality Python experience, including testing, maintainability, debugging, and release practices.
Experience creating evaluation methodologies, not only running established benchmarks.
Strong ability to investigate model or agent failures using traces, outputs, datasets, and experimental results.
Ability to work onsite in New York City five days per week.
Must already have unrestricted U.S. work authorization; visa sponsorship is not available.
Experience with PyTorch and Hugging Face / Transformers.
Experience with LLM evaluation and benchmarking, experiment design, and statistical or quantitative analysis.
Experience with evaluation datasets and labeling methodologies.
Experience with model/LLM-as-a-judge evaluation.
Hands-on experience with ML experimentation and debugging.

Tools and technical focus

Technologies: Python, PyTorch, Hugging Face, Transformers, Weights & Biases, Hydra, SQL, Kafka

Preferred: Weights & Biases, Hydra, SQL, Kafka or other streaming/event-driven infrastructure, evaluation/observability frameworks, agent simulation and multi-agent systems, large-scale data pipelines

Compensation

$220,000–$300,000 base salary + competitive equity
Final compensation depends on experience, technical depth, research contributions, and interview performance.

Benefits

Employee assistance program

Application questions

Can you work full-time on-site in New York City 5 days per week, and do you have unrestricted U.S. work authorization without requiring visa sponsorship?
Have you published ML/LLM research at a recognized conference/journal, or released substantial open-source research/ML work of comparable quality?
Do you have hands-on experience with PyTorch and Hugging Face/Transformers?
Do you have production-level Python experience (tests, maintaining production code, debugging, managing releases)?
How many years of professional experience do you have designing and owning LLM evaluations?
Have you personally designed, built, and run an LLM evaluation system end to end, including defining the evaluation methodology and metrics?
Please share your LinkedIn URL.

Hiring process (technical focus)

Designing an evaluation from scratch
Diagnosing an LLM or agent failure from traces
Building a production evaluation pipeline
Research methodology and experimental design
Python and ML engineering
Previous research or open-source work

View full job page →

Get Ml Experimentation Engineer Jobs alerts

New Ml Experimentation Engineer Jobs, daily.