AI Software Test Engineer

AI Software Test Engineer role in Ann Arbor focused on validating AI applications for reliability, accuracy, safety, and performance.

Responsibilities

• Develop and implement comprehensive testing strategies across AI applications, platforms, and services
• Validate AI model outputs for accuracy, consistency, reliability, and safety to support high-quality user experiences
• Design and execute functional, integration, end-to-end, regression, and performance testing for AI solutions
• Create test cases for prompt-driven, agentic, and retrieval-based AI workflows
• Validate AI guardrails, business rules, permissions, and governance controls to support responsible AI deployment
• Run adversarial, negative, and edge-case testing to identify potential failures and hallucinations
• Build and maintain automated test frameworks tailored for AI applications
• Develop automated evaluation pipelines for AI responses and workflows to enable continuous quality assessment
• Integrate AI testing into CI/CD pipelines for continuous quality and rapid iteration
• Implement automated quality scoring and regression detection mechanisms to maintain release standards
• Create reusable test data, mocks, simulators, and validation frameworks to improve efficiency and coverage
• Test AI agents, workflows, APIs, and tool-calling capabilities
• Validate integrations with external systems, diverse data sources, and enterprise services
• Verify performance, reliability, scalability, and resiliency of AI workloads
• Execute load and stress testing to confirm AI services can handle expected demand
• Collaborate with software engineers, AI engineers, product owners, architects, and security teams throughout delivery
• Participate in design reviews to provide early quality feedback
• Help define test strategy, establish quality standards, and promote best practices
• Support production readiness reviews and lead defect triage for timely resolution

Requirements

• Bachelor’s degree in Computer Science, Software Engineering, Information Systems, or a related technical field
• 3–6 years of progressive experience in software testing, QA automation, or quality engineering
• Experience developing automated test solutions using one or more: Python, Java, JavaScript/TypeScript, C#
• Demonstrated experience with API testing and automation (e.g., Postman)
• Strong foundation in test automation principles and best practices
• Working knowledge of SDLC and Agile methodologies
• Experience integrating testing into CI/CD (e.g., GitHub Actions, Jenkins)
• Experience testing distributed systems, web applications, and APIs
• Familiarity with Docker and Kubernetes
• Experience testing on cloud platforms (Azure, AWS, or GCP)
• Proficiency in quality engineering concepts including Defect Analysis and Root Cause Investigation

Technologies

• Python, Java, JavaScript/TypeScript, C#
• Postman
• GitHub Actions, Jenkins
• Docker, Kubernetes
• Azure, AWS, GCP
• Selenium, Playwright
• JUnit, PyTest

Benefits

• Choice of three major medical plans (including vision coverage) plus dental insurance
• Paid sick leave for all hourly talent in the U.S. and Canada
• 401(k) with a match and immediate vesting
• Exclusive employee discounts through Promo Code: Aquent
• Free online training through Aquent Gymnasium

Preferred Qualifications (Nice-to-Have)

• Experience testing generative AI applications, LLM-based systems, AI agents, or RAG applications
• Familiarity with AI models and platforms such as OpenAI, Claude, Gemini, or Azure OpenAI
• Experience building evaluation and benchmarking frameworks for AI solutions
• Knowledge of responsible AI principles, AI governance frameworks, and AI risk management practices
• Experience with AI testing techniques including Prompt Testing, Response Evaluation, Hallucination Detection, Agent Workflow Validation, RAG Validation, AI Safety Testing, AI Regression Testing, and AI Benchmarking
• Familiarity with test automation frameworks such as Selenium, Playwright, JUnit, or PyTest
• Exposure to quality engineering domains such as Performance Testing, Load Testing, and Security Testing

Location: Ann Arbor, MI (onsite)

Compensation: USD 42–46 per hourly

Experience level: Mid-level (3+ years)