Rag Benchmarking Engineer Jobs

1 job found

Today

AI Engineer (All Levels)

National Steel and Shipbuilding Com San Diego, CA

San Diego, CA

Today

$90k - $160k/yr Full time Agentic Ai Orchestration Ai Engineer Artificial Intelligence

Apply Now

Job Description

National Steel and Shipbuilding Com is seeking an AI Engineer to help bring production-ready AI capabilities to life within Azure Government (GCCH) environments. This onsite role in San Diego, CA focuses on implementing and evaluating LLM and RAG solutions, supporting agentic AI workflows, and enabling secure connectivity between AI agents and enterprise systems.

The position supports design, build, and operationalization of AI workloads across teams, with an emphasis on secure implementation practices, monitoring, and helping transition proofs of concept into production.

Key Responsibilities

Support testing and evaluation of LLMs and model variants (including Azure OpenAI and other approved models) for performance and quality.
Assist with RAG benchmarking across latency, relevance, accuracy, and cost, and help optimize extraction engines, embeddings, and prompt strategies.
Help build and configure MCP servers (or similar integrations) to securely connect AI agents to enterprise data and software.
Assist in implementing agentic AI workflows (multi-step, tool-using agents) using platforms such as N8N, Azure Foundry, and Copilot Studio.
Contribute to the design and implementation of modular AI microservices and APIs that can be reused across projects.
Work with BI and DevOps teams to adopt AI and ML capabilities and move proofs of concept toward production.
Support implementation of RBAC, secrets/key management (including Key Vault), and data protection for AI workloads.
Assist with monitoring, incident response, and documentation for AI workloads in Azure Gov environments.

Requirements

Bachelor’s degree in Computer Science, Information Technology, or related field (or equivalent education/experience).
Minimum 1–2 years experience with LLMs, RAG, or AI solution development.
Foundational understanding of Azure or other cloud platforms; Azure Gov exposure is a plus.
Interest or basic experience in prompt engineering, RAG architectures, and agentic AI workflows.
Exposure to RBAC, secrets/key management (Key Vault), and data protection concepts.
Strong organizational and verbal and written communication skills.
Ability to operate in a fast-paced, evolving environment and respond with urgency to IT situations.
This position may include other responsibilities and duties as assigned to support organizational security goals and initiatives.

Technologies

Azure Government (GCCH)
Azure OpenAI
MCP servers
N8N
Azure Foundry
Copilot Studio
RBAC
Key Vault

Benefits

Medical & dental insurance coverage
401(k) retirement savings plan
Flexible spending accounts/HSAs
Life and disability insurance
Company paid holidays and paid time off (PTO)
Wellness coaching
Onsite health services

Physical Demands

Work is generally performed in office conditions with good lighting.
Specific vision abilities required: close vision, distance vision, peripheral vision, depth perception, and ability to adjust focus.
Regularly required to sit, use hands to fingers, handle or feel objects or controls, and to talk and hear.
Must be physically able to walk to all yard locations.
May occasionally be exposed to working outdoors with a moderate noise level.

Salary: USD 90,300 - 159,900 per year (estimated range). Schedule: 1st shift. Security Clearance: No Clearance. Posting end date: Until Filled.

View full job page →

Get Rag Benchmarking Engineer Jobs alerts

New Rag Benchmarking Engineer Jobs, daily.