Rag Evaluation Engineer Jobs

1 job found

Today

AI Engineer

Alight Deerfield, IL 🇺🇸 Remote

Deerfield, IL

Today

$120k - $220k/yr Full time Agent Orchestration Agentic Systems Artificial Intelligence

Apply Now

Job Description

Alight is hiring an AI Engineer to build the AI engineering platform that supports AI delivery across Alight. This hands-on role focuses on developing production-ready AI agent capabilities, including orchestration patterns, retrieval, evaluation, ingestion pipelines, and operational deployment practices.

Key Responsibilities

Develop the agent patterns, retrieval, and guardrails behind agentic assistants that answer participant questions and perform transactions on their behalf.
Build and maintain an MCP server that provides a skills registry for AI coding agents inside the IDE.
Create reusable agent patterns in LangGraph, including multi-agent orchestration, memory, tool-use safety, and guardrails.
Develop an agent evaluation platform using frameworks such as DeepEval and Ragas, with quality gates that prevent deployment of inadequate changes in CI/CD.
Implement document ingestion and retrieval pipelines, including chunking, embedding, vector search, reranking, and citations.
Automate CI/CD and deployment for AI agents on AWS using ECS, Fargate, and Agent Core.
Deliver AI-enabled devcontainers with built-in tooling, validation, and testing.
Implement OpenTelemetry instrumentation, dashboards, and LLM cost and token tracking for production workloads.
Translate governance controls into pipeline checks with automated evidence collection.
Provide workload resilience with semantic caching, rate limiting, and graceful degradation under load.
Publish reference implementations plus SDK and CLI tooling and documentation used by other engineers.

Required Qualifications

Python proficiency (3.10+), as the primary language across the team.
AWS working knowledge of core services including IAM, ECS/Fargate, and networking, with comfort operating in cloud-native environments.
Software engineering discipline, including version control (Git/GitHub), code review, testing, documentation, and clean code practices.
Security awareness, including secure handling of sensitive data with a focus on secrets, permissions, and data protection.

Technology Stack

Python, AWS, IAM, ECS, Fargate
LangGraph, DeepEval, Ragas
OpenTelemetry, Agent Core, MCP
Git, GitHub, Terraform, CI/CD
Vector search, Reranking, Embeddings

Compensation

USD 120,000 - 220,000 per yearly

Work Location

Deerfield, IL (remote)

Benefits

Health, dental, and vision coverages starting Day One
Wellbeing programs
Retirement plans with contribution matching
Generous time off
Parental leave
Continuing education
Career growth opportunities

Flexible Working

Flexible working arrangements considered wherever possible.
Alight has been a leader in the flexible workspace and has been named “Top 100 Company for Remote Jobs” 6 years in a row.

Great Place to Work

Multiple awards of recognition, including “Great Place to Work” for the past 7 years.
Fortune’s “Best Companies to Work For.”

Relevant Expertise Areas

AI agent development: LangGraph or similar, multi-agent orchestration, memory, tool use, guardrails, prompt and workflow design.
Data and retrieval engineering: chunking, embeddings and vector search, RAG, hybrid search, reranking, retrieval evaluation.
Quality and evaluation: agent eval frameworks (DeepEval, Ragas), CI/CD quality gates, groundedness and safety metrics, test automation for AI systems.
Platform and infrastructure: CI/CD, Terraform, container orchestration on AWS (ECS, Fargate), secrets and IAM, progressive deployment.
Observability and operations: OpenTelemetry tracing, dashboards and alerting, LLM cost and token tracking, feedback loops.
Developer experience: MCP servers, devcontainers, CLI and SDK tooling, prompt libraries, documentation, and reference implementations built for other engineers.

Application and Interview Information

Background checks may include items such as SSN/SIN validation, education verification, employment verification, and criminal check, and may also include searches against global sanctions and government watch lists, credit check, and/or drug test depending on the position.
All virtual interviews must be conducted on video.
Alight has a camera-on culture and may require occasional travel to physical office locations.

Commitment to Inclusion

Reasonable accommodations will be provided for applicants with disabilities during the hiring process.
To request accommodation, contact alightcareers@alight.com.

Authorization to Work

Applicants must have work authorization that does not require visa sponsorship now or in the future for employment authorization in the Employing Country and with Alight.

Additional Note

This job description does not restrict management’s right to assign or reassign duties and responsibilities of this job to other entities, including subsidiaries, partners, or purchasers of Alight business units.

View full job page →

Get Rag Evaluation Engineer Jobs alerts

New Rag Evaluation Engineer Jobs, daily.