Llm As A Judge Engineer Jobs

1 job found

Today

AI Engineer / Generative AI Engineer

TMS LLC Los Angeles, CA

Los Angeles, CA

Today

Contract Agentic Ai Ai Agent Ai Evals

Apply Now

Job Description

The AI Engineer / Generative AI Engineer role at TMS LLC focuses on end-to-end delivery of production-grade AI agents, from implementation and deployment to ongoing monitoring and continuous improvement. The position is based in Los Angeles, CA with a hybrid work arrangement.

Role Overview

You will build, deploy, monitor, and iteratively enhance AI agents designed to operate reliably in production environments. The work emphasizes agent development using Python and Claude Agent SDK, integration with AWS Bedrock AgentCore components, and systematic evaluation and observability to support continuous quality improvement.

Key Responsibilities

Develop AI agents using Python and Claude Agent SDK.
Deploy agents using AWS Bedrock AgentCore Runtime.
Design golden datasets and implement LLM-as-a-Judge evaluations.
Integrate automated evaluations into CI/CD pipelines as quality gates.
Implement Langfuse and/or OpenTelemetry for AI observability and tracing.
Evaluate agent accuracy, task completion, tool execution, and performance.
Support RAG pipelines and AWS data processing workflows.

Required Qualifications

Experience with Python and agentic AI development.
Experience with AWS Bedrock AgentCore Evaluations / Bedrock evaluation.
Proficiency in golden-dataset design and curation.
Ability to design LLM-as-a-Judge approaches, including calibration against human labels.
Experience implementing automated evaluation and integrating evaluation checks into CI gates.
Hands-on experience with Langfuse and/or OpenTelemetry.
6–8+ years of software engineering, cloud engineering, or AI/ML experience (with the role targeting 15+ years minimum experience).
Hands-on experience working on production AI agents with continuous evaluation.
Strong problem-solving and troubleshooting skills.

Technologies and Tools

Python, Claude Agent SDK
AWS Bedrock AgentCore, AWS Bedrock AgentCore Runtime
LLM-as-a-Judge, CI/CD
Langfuse, OpenTelemetry
RAG, AWS data processing workflows
Bedrock evaluation

Good to Have

RAGAS or equivalent evaluation frameworks.
Trajectory evaluation and agentic red-teaming.
Bedrock Guardrails, Strands Agents SDK, and GitHub Actions.

Position Details

Location: Los Angeles, CA (hybrid)
Employment type: Contract
Experience level: 15+ years minimum

View full job page →

Get Llm As A Judge Engineer Jobs alerts

New Llm As A Judge Engineer Jobs, daily.