Machine Learning Engineer – Evals

Mango Global Technologies is hiring a Machine Learning Engineer – Evals in New York, NY (onsite) to own the end-to-end evaluation of an AI memory and identity system. This role focuses on building production-quality evaluation infrastructure, turning experiment results into actionable fixes, and continuously evolving evaluation definitions as models and product capabilities change.

What you’ll do

• Define how to determine whether evolving entity representations are improving.
• Build the infrastructure needed to measure representation improvements reliably.
• Use evaluation results to identify and fix failures across the evaluation loop.
• Define what “better” means for representations that evolve over time.
• Design novel evaluation methodologies when traditional ground truth is unavailable or insufficient.
• Develop scoring frameworks, metrics, rubrics, judge models, and diagnostic methodologies.
• Continuously evolve evaluation definitions as models and product capabilities shift.
• Build production-quality Python pipelines and evaluation harnesses.
• Manage datasets, labels, experiment versions, model versions, judges, reruns, and regression testing.
• Create infrastructure that helps researchers and engineers answer new evaluation questions quickly.
• Integrate experiment tracking and observability into evaluation workflows.
• Analyze model outputs, agent trajectories, and execution traces to find meaningful failure modes.
• Translate evaluation findings into concrete engineering or research improvements.
• Run experiments, compare results, document findings, and drive the resulting fixes.
• Develop and operate simulation environments that model large numbers of entities.
• Increase the throughput and reliability of evaluation runs.
• Enable rapid iteration across models, prompts, representations, and evaluation strategies.
• Own the full process from research question → evaluation design → data → implementation → execution → analysis → diagnosis → rerun → recommendation.

Required qualifications

• 3+ years of professional experience designing and owning LLM evaluations.
• Demonstrated experience designing, building, and running LLM evaluations end to end.
• Production-quality Python experience, including testing, maintainability, debugging, and release practices.
• Experience creating evaluation methodologies, not only running established benchmarks.
• Strong ability to investigate model or agent failures using traces, outputs, datasets, and experimental results.
• Ability to work onsite in New York City five days per week.
• Must already have unrestricted U.S. work authorization; visa sponsorship is not available.
• Experience with PyTorch and Hugging Face / Transformers.
• Experience with LLM evaluation and benchmarking, experiment design, and statistical or quantitative analysis.
• Experience with evaluation datasets and labeling methodologies.
• Experience with model/LLM-as-a-judge evaluation.
• Hands-on experience with ML experimentation and debugging.

Tools and technical focus

Technologies: Python, PyTorch, Hugging Face, Transformers, Weights & Biases, Hydra, SQL, Kafka

Preferred: Weights & Biases, Hydra, SQL, Kafka or other streaming/event-driven infrastructure, evaluation/observability frameworks, agent simulation and multi-agent systems, large-scale data pipelines

Compensation

• $220,000–$300,000 base salary + competitive equity
• Final compensation depends on experience, technical depth, research contributions, and interview performance.

Benefits

• Employee assistance program

Application questions

• Can you work full-time on-site in New York City 5 days per week, and do you have unrestricted U.S. work authorization without requiring visa sponsorship?
• Have you published ML/LLM research at a recognized conference/journal, or released substantial open-source research/ML work of comparable quality?
• Do you have hands-on experience with PyTorch and Hugging Face/Transformers?
• Do you have production-level Python experience (tests, maintaining production code, debugging, managing releases)?
• How many years of professional experience do you have designing and owning LLM evaluations?
• Have you personally designed, built, and run an LLM evaluation system end to end, including defining the evaluation methodology and metrics?
• Please share your LinkedIn URL.

Hiring process (technical focus)

• Designing an evaluation from scratch
• Diagnosing an LLM or agent failure from traces
• Building a production evaluation pipeline
• Research methodology and experimental design
• Python and ML engineering
• Previous research or open-source work