EngineerJobs.io
← Back to all jobs

Job Description

The Sr/Principal AI Engineer will help design and deliver agentic, production-ready AI services that use LLM orchestration, multi-agent coordination, and RAG systems to improve the workers’ compensation experience. This role operates in a highly governed PHI environment and emphasizes evaluation, safety, and GenAIOps practices.

Location and Work Arrangement

  • Location: United States (remote)
  • Work setup: Performed in the colleague’s home, with the ability to set up computer equipment in a home office
  • In-person meetings: Occasional team meetings may be required depending on the position

Compensation

  • Salary: USD 120,442 - 192,707 per year

Responsibilities

  • Translate product specifications into AI service architectures, decomposing business intent into agent workflows, data source integrations, and scalable service design
  • Build agentic AI workflows using Azure AI Foundry Agent Service and the Microsoft Agent Framework, including multi-agent coordination, tool integrations, and agent lifecycle management
  • Develop and iterate on LLM-based solutions, including prompt engineering, model selection, and cost or token optimization, as well as RAG pipeline design
  • Build evaluation systems using rubrics, golden datasets, and judge agents to validate agent correctness and safety before production deployment
  • Contribute to agentic systems with engineering practices focused on separation of concerns, memory architecture, and tool integration
  • Support end-to-end delivery across model experimentation, evaluation design, and production system deployment
  • Build and maintain APIs, microservices, CI/CD pipelines, and cloud-native deployments with attention to observability and reliability
  • Support GenAIOps processes for automated testing, regression evaluation, latency monitoring, and continual improvement
  • Balance performance, safety, responsible AI principles, and cost in system design, including guardrails, fallbacks, red-teaming strategies, and human-in-the-loop (HITL) workflows
  • Partner with legal and ethics teams to align with OneCall’s Responsible AO standards in a highly governed PHI space
  • Contribute to automated evaluation suites integrating user signals, continual learning cycles, and ongoing model updates
  • Design evaluation frameworks covering hallucination rate and factual consistency, relevance and precision/recall, latency and throughput, system-level performance, and cost tracking and efficiency
  • Collaborate with product, engineering, data science, security, ethics, and infrastructure teams to build robust, compliant AI systems
  • Communicate technical findings clearly to cross-functional stakeholders and, where relevant, clients
  • Share knowledge with peers and support a culture of technical learning across RAG, agents, prompt engineering, and AI safety

Required Qualifications

  • Experience: 5+ years in software engineering, traditional ML engineering, or applied AI with hands-on building responsibilities
  • Proven experience shipping generative AI features or systems end-to-end, from prototyping through production
  • Working knowledge of evaluation design, experimentation frameworks, and data labeling strategies for LLM applications
  • Experience with RAG architectures (vector-based retrieval; exposure to hybrid or graph-based approaches is a plus)
  • Experience with agentic patterns (tool use, routing, memory management; multi-agent systems experience is a plus)
  • Experience with ReAct, RLAIF, and other HITL + feedback loop approaches
  • Experience with AI-specific tools and frameworks, including orchestration frameworks, vector and graph databases, and Model +API ecosystems
  • Strong background in system design, architecture, and production-grade deployment
  • Familiarity with Azure cloud services, logging, monitoring, and observability patterns
  • Solid prompt engineering and context engineering skills; familiarity with conversation design principles
  • Clear communication skills and the ability to explain technical concepts to non-technical collaborators and stakeholders
  • Ability to collaborate effectively across engineering, product, and data science teams; experience supporting or informally mentoring peers is a plus
  • Familiarity with cost and latency tradeoffs for LLM workloads
  • Comfort working in high-ambiguity environments with cross-functional teams
  • Eagerness to grow into a technical leadership role and support junior team members
  • Ability to work independently and within a team environment
  • Alignment with the company’s diversity, equity, inclusion, and belonging standards
  • Demonstrated ability to support core values: Think Big, Go Fast, Deliver Awe, Win Together, and Care Deeply

Preferred Qualifications

  • Experience in healthcare, workers compensation, claims processing, or other regulated industries
  • Experience in client-facing consulting or enterprise transformation environments

Technology Stack

  • Azure AI Foundry Agent Service
  • Microsoft Agent Framework
  • LLM, RAG
  • CI/CD
  • APIs, microservices
  • GenAIOps
  • HITL
  • ReAct, RLAIF
  • Azure cloud services
  • Vector databases, graph databases
  • Model +API ecosystems
  • PHI

Physical / Mental Demands and Work Environment

  • Primarily sedentary work
  • Ability to remain stationary for extended periods and operate a computer and other office equipment
  • Ability to assess information and files stored electronically and converse or exchange accurate information with others
  • Ability to discern text displayed on a monitor, input data using a keyboard, and adjust focus to distances up to three feet
  • Ability to learn new tasks, follow established processes, maintain focus, complete tasks independently, and handle multiple tasks simultaneously
  • Ability to communicate professionally with colleagues and customers and complete tasks with speed or productivity requirements
  • Ability to manage moderate to significant mental stress due to a dynamic and heavy workload

Similar Jobs