Sr AI Engineer
Job Description
The Sr/Principal AI Engineer will help design and deliver agentic, production-ready AI services that use LLM orchestration, multi-agent coordination, and RAG systems to improve the workers’ compensation experience. This role operates in a highly governed PHI environment and emphasizes evaluation, safety, and GenAIOps practices.
Location and Work Arrangement
- Location: United States (remote)
- Work setup: Performed in the colleague’s home, with the ability to set up computer equipment in a home office
- In-person meetings: Occasional team meetings may be required depending on the position
Compensation
- Salary: USD 120,442 - 192,707 per year
Responsibilities
- Translate product specifications into AI service architectures, decomposing business intent into agent workflows, data source integrations, and scalable service design
- Build agentic AI workflows using Azure AI Foundry Agent Service and the Microsoft Agent Framework, including multi-agent coordination, tool integrations, and agent lifecycle management
- Develop and iterate on LLM-based solutions, including prompt engineering, model selection, and cost or token optimization, as well as RAG pipeline design
- Build evaluation systems using rubrics, golden datasets, and judge agents to validate agent correctness and safety before production deployment
- Contribute to agentic systems with engineering practices focused on separation of concerns, memory architecture, and tool integration
- Support end-to-end delivery across model experimentation, evaluation design, and production system deployment
- Build and maintain APIs, microservices, CI/CD pipelines, and cloud-native deployments with attention to observability and reliability
- Support GenAIOps processes for automated testing, regression evaluation, latency monitoring, and continual improvement
- Balance performance, safety, responsible AI principles, and cost in system design, including guardrails, fallbacks, red-teaming strategies, and human-in-the-loop (HITL) workflows
- Partner with legal and ethics teams to align with OneCall’s Responsible AO standards in a highly governed PHI space
- Contribute to automated evaluation suites integrating user signals, continual learning cycles, and ongoing model updates
- Design evaluation frameworks covering hallucination rate and factual consistency, relevance and precision/recall, latency and throughput, system-level performance, and cost tracking and efficiency
- Collaborate with product, engineering, data science, security, ethics, and infrastructure teams to build robust, compliant AI systems
- Communicate technical findings clearly to cross-functional stakeholders and, where relevant, clients
- Share knowledge with peers and support a culture of technical learning across RAG, agents, prompt engineering, and AI safety
Required Qualifications
- Experience: 5+ years in software engineering, traditional ML engineering, or applied AI with hands-on building responsibilities
- Proven experience shipping generative AI features or systems end-to-end, from prototyping through production
- Working knowledge of evaluation design, experimentation frameworks, and data labeling strategies for LLM applications
- Experience with RAG architectures (vector-based retrieval; exposure to hybrid or graph-based approaches is a plus)
- Experience with agentic patterns (tool use, routing, memory management; multi-agent systems experience is a plus)
- Experience with ReAct, RLAIF, and other HITL + feedback loop approaches
- Experience with AI-specific tools and frameworks, including orchestration frameworks, vector and graph databases, and Model +API ecosystems
- Strong background in system design, architecture, and production-grade deployment
- Familiarity with Azure cloud services, logging, monitoring, and observability patterns
- Solid prompt engineering and context engineering skills; familiarity with conversation design principles
- Clear communication skills and the ability to explain technical concepts to non-technical collaborators and stakeholders
- Ability to collaborate effectively across engineering, product, and data science teams; experience supporting or informally mentoring peers is a plus
- Familiarity with cost and latency tradeoffs for LLM workloads
- Comfort working in high-ambiguity environments with cross-functional teams
- Eagerness to grow into a technical leadership role and support junior team members
- Ability to work independently and within a team environment
- Alignment with the company’s diversity, equity, inclusion, and belonging standards
- Demonstrated ability to support core values: Think Big, Go Fast, Deliver Awe, Win Together, and Care Deeply
Preferred Qualifications
- Experience in healthcare, workers compensation, claims processing, or other regulated industries
- Experience in client-facing consulting or enterprise transformation environments
Technology Stack
- Azure AI Foundry Agent Service
- Microsoft Agent Framework
- LLM, RAG
- CI/CD
- APIs, microservices
- GenAIOps
- HITL
- ReAct, RLAIF
- Azure cloud services
- Vector databases, graph databases
- Model +API ecosystems
- PHI
Physical / Mental Demands and Work Environment
- Primarily sedentary work
- Ability to remain stationary for extended periods and operate a computer and other office equipment
- Ability to assess information and files stored electronically and converse or exchange accurate information with others
- Ability to discern text displayed on a monitor, input data using a keyboard, and adjust focus to distances up to three feet
- Ability to learn new tasks, follow established processes, maintain focus, complete tasks independently, and handle multiple tasks simultaneously
- Ability to communicate professionally with colleagues and customers and complete tasks with speed or productivity requirements
- Ability to manage moderate to significant mental stress due to a dynamic and heavy workload