AI Engineer, Evaluation

Based in New York, NY on a hybrid schedule, Distyl AI offers a collaborative, purpose-driven environment where engineers can shape production AI systems. This role delivers a competitive compensation package, meaningful equity, and a broad benefits offering. You will work with state-of-the-art AI models on real business problems, with fully covered medical, dental, and vision for you and your dependents, flexible time off, retirement planning resources, wellness programs including fertility and family-building benefits through Carrot, and in-office meals that support a productive cadence.

Responsibilities

• Design and implement evaluation frameworks that enable Evaluation-Driven Development for AI systems deployed in customer environments
• Define how system quality is measured in each domain, ensuring evaluation signals reflect real user needs, domain constraints, and business objectives
• Build and maintain golden test cases and regression suites in Python, using both human-authored and AI-assisted test generation to capture critical behaviors and edge cases. These test suites are treated as first-class system components that evolve alongside the AI system itself
• Develop and maintain evaluation pipelines offline and online that integrate into system iteration loops. Evaluation results inform prompt design, agent logic, model selection, and release readiness to drive measurable improvements
• Define, calibrate, and operate LLM-based graders, aligning automated judgments with expert human assessments. They investigate where evaluation signals diverge from real-world outcomes and refine grading approaches to maintain signal quality as systems evolve
• Collaborate closely with Forward Deployed AI Engineers, Architects, Product Engineers, AI Strategists, and domain experts to ensure evaluation frameworks meaningfully guide production development and deployment

Requirements

• 2+ years of software engineering experience
• Strong Python engineering skills: write clean, maintainable Python and build evaluation and experimentation pipelines that run in production; treat evaluation code with the same rigor as application code
• Experience with Evaluation-Driven or Experiment-Driven Development: using structured evaluation or experimentation frameworks to drive system iteration and avoiding overfitting to metrics that do not reflect real outcomes
• Ability to translate human judgment into code: collaborate with subject matter experts to elicit high-quality judgments and encode them into test cases, scoring functions, and graders that scale
• Systems-oriented mindset: understand how evaluation interacts with prompts, agents, data, and deployment. design evaluation systems that support fast iteration while maintaining trust and safety in production
• AI-native working style: leverage AI tools to generate tests, analyze failures, explore edge cases, and accelerate debugging and iteration
• Travel: 10 to 50 percent of the time, depending on project needs and role

Technologies

• Python

Benefits

• Base salary range of $150,000 to $250,000 per year, plus meaningful equity and comprehensive benefits
• 100% coverage of medical, dental, and vision insurance for you and dependents
• Flexible time off
• Retirement and financial planning resources, including pre-tax HSA, FSA, commuter accounts, 401(k), and financial coaching
• Comprehensive wellness benefits, including physical and mental well-being, plus fertility and family-building benefits through Carrot
• Complimentary in-office lunches and snacks
• Access to state-of-the-art AI models and tools for real-world business challenges
• Ownership of high-impact projects across leading enterprises
• Mission-driven, fast-moving culture that values curiosity, pragmatism, and excellence