Llm Testing Engineer Jobs

1 job found

Today

AI Engineer

Hollstadt Consulting Minneapolis, MN 🇺🇸 Remote

Minneapolis, MN

Today

$105k - $140k/yr Full time AI Ai Engineer Ai Engineering

Apply Now

Job Description

Hollstadt Consulting is building out its AI Center of Excellence and needs an AI/ML Engineer to support cross-functional teams with production-focused machine learning, AI, Generative AI, and agentic AI solutions. This role is centered on designing and delivering end-to-end capabilities, from pipeline development to testing and secure integration across enterprise environments. This position is remote in Minnesota, with local preference.

Key work includes building production AI/ML pipelines and GenAI or agent capabilities tailored to domain-specific needs. The role supports automation of complex document understanding and structured content extraction, while also establishing reliability measures such as LLM testing frameworks and regression suites for drift and prompt breakage.

Responsibilities

Collaborate across GenAI platforms and libraries, including AWS, Salesforce, Oracle, Snowflake, MS Copilot, and other 3rd party GenAI platforms.
Automate workflows to extract highly accurate structured outputs from complex, multimodal unstructured content using tools such as AWS Textract and Bedrock.
Design and build Model Context Protocol (MCP) hosts, clients, and servers.
Establish and apply frameworks for automated LLM testing.
Create regression test suites to detect drift and prompt breakage.
Integrate with internal and external web services using secure authentication and authorization mechanisms.
Adopt safe practices to protect against prompt injections and jailbreaks, aligning with enterprise security guidelines.
Design, develop, and deploy production-grade traditional ML models including regression, classification, clustering, and recommender systems.
Design, maintain, and optimize end-to-end AI/ML pipelines across data ingestion, training, evaluation, deployment, and monitoring on cloud infrastructure such as AWS (or equivalent).
Ensure AI/ML solutions are scalable, reliable, secure, and cost-effective in cloud environments.
Create reusable components, frameworks, and best practices to accelerate AI development.
Design and develop GenAI solutions using prompt engineering, Context Engineering, Retrieval-Augmented Generation (RAG), and custom pipelines.
Design and develop interoperable AI agents using MCP and/or Google A2A.
Partner with data scientists, architects, product managers, business stakeholders, and technical teams to align AI solutions with organizational goals.
Provide hands-on technical support and mentorship to technical teams across the enterprise.

Requirements

3+ years of experience designing and deploying ML/AI solutions in real-world environments.
Very strong Python skills.
Strong hands-on experience with LLM APIs (including OpenAI, Azure OpenAI, Gemini, Anthropic) using Python and Python-based frameworks.
Strong hands-on experience with prompt engineering, context construction, and grounding strategies.
Strong hands-on experience with RAG workflows, including extracting, chunking, and creating embeddings from unstructured documents from sources such as O365 (email, Word, Excel), PDFs, and webpages.
Comfort building MCP clients, servers, and hosts.
Expertise building REST APIs and integrating with internal and external APIs.
Hands-on experience with Intelligent Document Processing and/or OCR technologies on complex documents.
Knowledge of Google A2A.
Deep experience in AWS including Lambda, Bedrock, Step Functions, API Gateway, and IAM.
Strong experience with observability tools such as Dynatrace or similar GenAI observability tools.
Excellent GenAI foundations and concepts, including enterprise data privacy, AI governance, and observability.
Proficiency in Python and common ML/AI libraries such as TensorFlow, PyTorch, and scikit-learn.
Strong understanding of data engineering, SQL, and feature engineering.
Hands-on experience with cloud services including AWS SageMaker, Lambda, ECS, S3, and IAM.
Familiarity with containerization (Docker) and orchestration (e.g., Airflow, Kubeflow).
Working knowledge of version control and collaboration tools such as Git, Jira, and Confluence.
Bachelor’s degree in computer science, engineering, or a related field.
Knowledge of machine learning algorithms, deep learning frameworks, cloud AI technologies, GenAI technologies, and emerging agentic AI technologies.
Knowledge of cloud platforms (AWS, Azure, GCP) for scalable AI/ML development.
Knowledge of responsible AI principles including bias mitigation and ethical deployment.
Knowledge of ML Ops best practices including CI/CD for ML, model monitoring, and versioning.
Ability to build robust, scalable, and efficient AI/ML solutions in cloud-native environments.
Ability to translate ambiguous business problems into clear technical ML/AI tasks.
Ability to communicate complex ideas clearly to technical and non-technical stakeholders.
Ability to learn and adapt quickly to emerging AI technologies, techniques, and tools.

Technologies

AWS, SalesForce, Oracle, Snowflake, MS Copilot
AWS Textract, Bedrock, Model Context Protocol (MCP), Google A2A
OpenAI, Azure OpenAI, Gemini, Anthropic
REST APIs, Dynatrace
TensorFlow, PyTorch, scikit-learn
O365, PDFs, webpages
Intelligent Document Processing, OCR
AWS Lambda, AWS Step Functions, AWS API Gateway, AWS IAM, AWS Sagemaker, AWS ECS, AWS S3
Docker, Airflow, Kubeflow
Git, Jira, Confluence, CI/CD, ML Ops
Retrieval-Augmented Generation (RAG), prompt engineering, Context Engineering, grounding strategies, embeddings

Compensation & Logistics

Location: Minnesota (remote); local preferred but not required
Salary: USD 105,000 - 140,000 per year, plus 8% AIP eligibility
Start date: 9/28/2026

Benefits

Comprehensive Benefit Plan including medical, dental, vision, life insurance, short-term disability, long-term disability, paid sick leave, and retirement benefits
401(k) + Matching: match on the first 4% of contributions
Bonus opportunities including Longevity Award bonus and referral bonus
Professional development with on-demand training through a consultant portal, including upskilling in Artificial Intelligence (AI)
Ongoing support and networking through a Consultant Coach program

Preferred Qualifications

Master’s in a related technical field
Hands-on experience with agentic AI frameworks
Prior contributions to open-source AI/ML projects or published research
AI/ML certifications from cloud providers
Experience in highly regulated industries (e.g., healthcare, finance) a plus

View full job page →

Get Llm Testing Engineer Jobs alerts

New Llm Testing Engineer Jobs, daily.