Model & Prompt Evaluation

Model & Prompt Evaluation Engineer Jobs

2 jobs found

Today

Senior Applied AI Engineer

Curai Remote 🌍 Remote

Remote

Today

$175k - $200k/yr Full time Senior Artificial Intelligence Fine Tuning

Apply Now

Job Description

Curai is looking for a Senior Applied AI Engineer to help design, build, and ship machine learning and LLM systems that influence how clinicians and patients engage with its healthcare platform. This role owns work end-to-end, from framing the right AI problem with partners to productionizing inference and validating real-world clinical and product impact.

What you’ll do

Lead technical execution for complex AI initiatives, owning the design and delivery of solutions within a product or technical domain and partnering with senior engineers on broader architecture.
Design, build, train, evaluate, and improve advanced machine learning and LLM-based systems for patient and provider-facing products, including areas such as conversational AI, personalization, user understanding, clinical decision support, and chronic care management.
Own problems end-to-end: scope with clinicians and product partners, build datasets and evaluations, iterate on modeling, and ship to production with appropriate monitoring and guardrails.
Build robust evaluation frameworks, including offline benchmarks, human-in-the-loop review, and online experiments, to ensure models are safe, accurate, and improving over time.
Develop and enhance the team’s platform to move quickly, including data pipelines, training and inference infrastructure, prompt and model management, and tooling for clinical reviewers.
Collaborate closely with clinicians, product, and engineering to translate medical and operational requirements into ML problems and deliver measurable improvements to patient and clinician experience.
Set technical direction for your area, mentor other engineers, and raise engineering and scientific rigor, with leadership scope scaling with seniority.
Stay current with the literature and the rapidly evolving AI ecosystem, bringing back what is most useful for patients and the team.

Requirements

Bachelor’s degree in Computer Science, Software Engineering, Math, or another related technical degree.
3+ years of hands-on engineering experience, including 1+ years building and deploying machine learning systems with generative AI (LLMS) and a clear track record of impact.
Strong software engineering fundamentals, with the ability to ship reliable, well-tested code in Python (or a comparable language) in production environments.
Practical understanding of modern LLM techniques, including prompting, retrieval-augmented generation, fine-tuning, and evaluation, along with trade-offs.
Comfort working with messy, real-world data and designing evaluations to determine whether a system is truly working.
Strong written and verbal communication, with the ability to collaborate across disciplines including clinicians, product managers, and engineers.
Bias toward action and ownership: driving ambiguous problems to results and helping others get there.
Care for the mission, with motivation to translate work into better health outcomes for real patients.

Technologies

Python
LLMs
Retrieval-augmented generation
Prompting
Fine-tuning
Evaluation

Nice to have

Experience applying ML or LLMs in healthcare, life sciences, or another regulated, high-stakes domain.
Experience with clinical NLP, medical knowledge representation, or working with electronic health record data.
Experience building agentic systems and tool-using LLMs in production.
Experience scaling ML infrastructure, including training pipelines, distributed inference, and evaluation platforms for a small, fast-moving team.
Track record of technical leadership, such as setting direction across teams, mentoring engineers, or publishing influential work.

Compensation and location

Location: Remote (remote) across the U.S.
Base salary: USD 175,000 - 200,000 per year
Actual base salary depends on qualifications and years of relevant experience.

Benefits

High-ownership work on problems that matter, with a tight feedback loop from real clinicians and patients.
A small, senior team where work shows up in the product quickly.
Competitive compensation, meaningful equity, and comprehensive benefits.
Remote-first, flexible work environment across the U.S.
Comprehensive medical, dental, and vision coverage.
Flexible spending plans.
Generous and flexible Paid Time Off (PTO), floating holidays, and parental leave.
401k plan with employer matching.
100% remote, work from home.

View full job page →

1d ago

Senior AI Engineer

Mitratech Remote 🌍 Remote

Remote

1d ago

$170k - $190k/yr Full time Backend Developer Senior Agentic Ai

Apply Now

Job Description

Mitratech is building an AWS-native GenAI platform that powers intelligent agent systems for both internal teams and customer-facing workflows. As a Senior AI Engineer, you will design, build, and operate agent services end-to-end, covering everything from architecture decisions to stable production operations.

This role focuses on owning agent capabilities on AWS, including the APIs, evaluation pipelines, and operational practices that keep reliability and cost in balance as the platform evolves.

What you will do

Design and own LLM-powered agents end-to-end for internal tooling, customer-facing features, and workflow automation, using Amazon Bedrock with LangChain/LangGraph and the langchain-aws, langchain-anthropic, and langchain-core stack
Architect agent workflows and RAG for team-level reuse, shaping design choices beyond your own services
Build and operate MCP servers that expose company data and services as structured tools for AI models
Define testing, reliability, and operational practices for your scope, owning delivery and on-call quality for major services
Stand up evaluation pipelines using Langfuse and evaluation harnesses, and apply prompt-versioning practices to improve reliability and drive cost and quality improvements
Lead design reviews, mentor engineers, and improve engineering practices across the team
Collaborate with product, data, and platform teams to scope and deliver agent-based solutions while applying platform patterns and governance hooks

What you bring

5+ years of software engineering experience, including multiple years building and operating LLM or AI agent systems in production
Expert-level Python and service development, including async, FastAPI, and Pydantic v2, with a track record of owning services end-to-end
Deep GenAI/LLM application experience, with strong RAG architecture and agent workflow design; experience with Claude via Amazon Bedrock or the Anthropic API is ideal
Strong AWS engineering experience, including Bedrock, Lambda, ECS, and API Gateway
Hands-on agent tooling, including LangChain/LangGraph and MCP (building and operating tool/resource servers)
Experience with Terraform and IaC, plus containerized delivery via CI/CD
Solid data-layer experience using systems such as Redis, Snowflake, DynamoDB, and Postgres/Aurora, with async SQLAlchemy
Identity and application security fundamentals, including JWT/OAuth and secrets management
Production observability and LLM evaluation experience, including tracing and evaluation metrics for latency, cost, and quality
Demonstrated ability to mentor engineers, lead design reviews, and establish engineering practices

Technologies you will work with

AWS-native GenAI platform, Amazon Bedrock, LangChain, LangGraph, langchain-aws, langchain-anthropic, langchain-core
FastAPI, Terraform, MCP, Langfuse, Python, Pydantic v2, async
Lambda, ECS, API Gateway, CI/CD
Redis, Snowflake, DynamoDB, Postgres/Aurora, async SQLAlchemy
JWT, OAuth, secrets management, Claude, Anthropic API
tracing and evals with latency/cost/quality metrics