Senior Machine Learning Engineer, Agentic Evals and Self-Improvement
Agentic Ai
Ai Agent
Ai Agent Platform
Ai Ml
Ai Platform
Artificial Intelligence
Artificial Intelligence Engineer
Data Analysis
Data Engineer
Data Pipeline
Data Processing
Data Science
Gen Ai Evaluation
Generative AI
Generative Ai Agents
Generative Ai Engineering
Machine Learning Engineering
Machine Learning Evaluation
Machine Learning Optimization
Machine Learning Platform
Machine Learning Systems
Machine Learning Training
Programming
Software Development
Software Engineering
Job Description
Iterable Nova is hiring a Senior Machine Learning Engineer to drive agentic evaluations and closed-loop improvements for production models.
Responsibilities
- Design and build evaluation systems for models and agents, including benchmarks, LLM judges, simulations, trajectory analysis, and regression tests
- Establish an end-to-end loop that moves from diagnosis to fix to measured improvement
- Translate evaluation outputs into automatic optimization for prompts, context, tools, rubrics, and agent harnesses
- Build synthetic data pipelines for evaluation and post-training, with focus on quality, privacy, and reproducibility
- Apply recent research methods in production, including LLM-as-judge, reward modeling, test-time search, and post-training
- Measure model and agent performance, identify failure modes, and use findings to drive automated improvements across prompts, context, tools, and harnesses
- Partner with Nova retrieval and product teams to connect real interactions to measurable improvements
- Mentor engineers and help define the technical direction for Nova
Requirements
- 5+ years hands-on MLE experience shipping complex ML systems into production at scale (examples: production ML pipelines or large-scale experimentation platforms)
- Master’s or PhD in Computer Science, Machine Learning, AI, or a related field
- Stay current on research and focus on shipping reliable changes that measurably improve customer experience
- Strong Python skills with emphasis on readable, modular, maintainable code, plus experience with modern ML frameworks
- Deep experience evaluating LLMs or agents, including multi-turn behavior, tool use, and reasoning
- Experience building automated evaluation systems such as LLM judges, reward models, or benchmark infrastructure
- Experience with at least one of: prompt or context optimization, post-training, preference optimization, or reinforcement learning
- Strong communication and cross-team collaboration
- Experience using LLM APIs from multiple providers (Anthropic, OpenAI, Google), including tool calling and prompt caching
- Experience with automatic prompt optimization and error analysis
Technologies
- Python
- LLM-as-judge
- Reward modeling
- Test-time search
- Post-training
- Anthropic
- OpenAI
Location
- San Francisco, CA (onsite)
- US
Compensation
- USD 150,000 - 250,000 per year
Bonus Points
- Experience with LLM APIs from multiple providers (Anthropic, OpenAI, Google), including tool calling and prompt caching
- Experience with automatic prompt optimization and error analysis
Similar Jobs
M
