EngineerJobs.io
← Back to all jobs
TMS LLC

AI Engineer / Generative AI Engineer

TMS LLC
Los Angeles, CA Contract Posted 4d ago
Agentic Ai Ai Agent Ai Evals Ai Observability Artificial Intelligence Artificial Intelligence Engineer Data Processing Genai Generative AI Llm As A Judge Rag Architectures Software Development
Apply Now

Job Description

The AI Engineer / Generative AI Engineer role at TMS LLC focuses on end-to-end delivery of production-grade AI agents, from implementation and deployment to ongoing monitoring and continuous improvement. The position is based in Los Angeles, CA with a hybrid work arrangement.

Role Overview

You will build, deploy, monitor, and iteratively enhance AI agents designed to operate reliably in production environments. The work emphasizes agent development using Python and Claude Agent SDK, integration with AWS Bedrock AgentCore components, and systematic evaluation and observability to support continuous quality improvement.

Key Responsibilities

  • Develop AI agents using Python and Claude Agent SDK.
  • Deploy agents using AWS Bedrock AgentCore Runtime.
  • Design golden datasets and implement LLM-as-a-Judge evaluations.
  • Integrate automated evaluations into CI/CD pipelines as quality gates.
  • Implement Langfuse and/or OpenTelemetry for AI observability and tracing.
  • Evaluate agent accuracy, task completion, tool execution, and performance.
  • Support RAG pipelines and AWS data processing workflows.

Required Qualifications

  • Experience with Python and agentic AI development.
  • Experience with AWS Bedrock AgentCore Evaluations / Bedrock evaluation.
  • Proficiency in golden-dataset design and curation.
  • Ability to design LLM-as-a-Judge approaches, including calibration against human labels.
  • Experience implementing automated evaluation and integrating evaluation checks into CI gates.
  • Hands-on experience with Langfuse and/or OpenTelemetry.
  • 6–8+ years of software engineering, cloud engineering, or AI/ML experience (with the role targeting 15+ years minimum experience).
  • Hands-on experience working on production AI agents with continuous evaluation.
  • Strong problem-solving and troubleshooting skills.

Technologies and Tools

  • Python, Claude Agent SDK
  • AWS Bedrock AgentCore, AWS Bedrock AgentCore Runtime
  • LLM-as-a-Judge, CI/CD
  • Langfuse, OpenTelemetry
  • RAG, AWS data processing workflows
  • Bedrock evaluation

Good to Have

  • RAGAS or equivalent evaluation frameworks.
  • Trajectory evaluation and agentic red-teaming.
  • Bedrock Guardrails, Strands Agents SDK, and GitHub Actions.

Position Details

  • Location: Los Angeles, CA (hybrid)
  • Employment type: Contract
  • Experience level: 15+ years minimum

Similar Jobs