AI Engineer

What you’ll gain

Join Vytalize Health in Kansas for an onsite AI Engineer role focused on designing, building, and maintaining agentic systems and LLM powered healthcare applications. You will contribute to automating workflows and delivering data driven clinical solutions with a strong emphasis on validation, reliability, and regulatory compliance. Collaborate across data, platform, product, and clinical teams to scale AI driven automation and elevate how care teams operate.

Responsibilities

• Design, build, and maintain agentic systems and LLM powered applications that automate healthcare workflows, data pipelines, and clinical decision support from concept to production.
• Create and orchestrate agents using LLM APIs (OpenAI, Anthropic, etc.) and agent frameworks (LangChain, LangGraph, CrewAI, or custom orchestration) to tackle complex multi step healthcare challenges.
• Develop prompt libraries, agent instructions, and reusable skills to improve accuracy, consistency, and reliability across use cases and data domains.
• Implement validation and confidence scoring layers to flag low confidence agent decisions for human review prior to production; establish guardrails and review workflows for agent authored code and outputs.
• Own end to end delivery of AI automated systems from scoping and requirements through development, testing, and validated deployment.
• Establish rigorous evaluation and QA frameworks for agentic systems, including golden datasets, test scenarios, output validation, hallucination detection, and regression testing.
• Define and maintain evaluation metrics for agent performance, reliability, and clinical appropriateness, tracking accuracy, hallucination rates, clinical validity, and real world impact.
• Set up observability, evaluation, and regression testing for agentic systems including decision tracing, lineage logging, and performance monitoring.
• Collaborate with data engineering and platform teams to integrate agent generated outputs such as dbt models, transformation logic, and recommendations into existing data architectures and clinical workflows.
• Ensure agentic systems comply with healthcare regulations such as HIPAA and FDA AI/ML guidance and uphold responsible AI practices including explainability, auditability, and clinician trust.
• Continuously evaluate new LLM models, agent frameworks, prompting techniques, and tooling, advising on adoption or migration based on healthcare needs like accuracy, cost, latency, and regulatory alignment.
• Partner with data engineering to implement robust data and input validation layers for agents, recognizing that data quality drives agent performance.
• Lead experiments to measure the impact of AI automation on speed, quality, compliance, and cost across healthcare workflows.
• Document agent architectures, prompting strategies, evaluation frameworks, and best practices for both technical and non technical stakeholders.
• Mentor AI Connector Engineers and others on agentic development patterns, LLM powered application design, and responsible AI practices.
• Provide on call support for production agent systems to troubleshoot issues and respond to performance concerns or hallucination signals.

Requirements

• 3+ years of professional experience in data engineering, backend engineering, machine learning, or a related field
• At least 1 year of hands on work building with LLM APIs and agent orchestration frameworks — not merely using AI coding assistants but designing agentic systems
• Strong Python and SQL skills
• Experience with cloud data platforms such as AWS and Databricks
• Solid grounding in data modeling, ETL/ELT patterns, and medallion architectures Bronze/Silver/Gold
• Experience creating and consuming APIs
• Proven experience with prompt engineering, agent evaluation, and validating LLM outputs
• Experience designing evaluation frameworks, test cases, and QA for AI/ML systems
• Ability to measure and track AI system performance using metrics and KPIs such as accuracy, precision, recall, and hallucination rates
• Strong debugging and analytical skills, especially in ambiguous or novel technical situations
• Excellent written and verbal communication, documenting agent reasoning, decisions and limitations clearly for broad audiences
• Comfortable in a fast paced environment with evolving AI capabilities and incomplete information

Technologies

• Python
• SQL
• AWS
• Databricks
• OpenAI
• Anthropic
• LangChain
• LangGraph
• CrewAI
• dbt
• Airflow
• Databricks Workflows
• LangSmith
• Langfuse
• FHIR
• HL7

Role at a glance

As an AI Engineer at Vytalize Health, you will design, build, and maintain agentic systems and LLM powered healthcare applications to automate complex workflows and accelerate data driven clinical solutions. You will orchestrate data retrieval, model inference, clinical logic, and tool use to tackle problems that typically require manual effort or specialized expertise. You will collaborate with data engineering, platform, product, and clinical teams to identify high impact opportunities for AI automation, from data source onboarding to decision support and evidence synthesis. Your focus is on production grade agentic systems with robust validation, clear confidence scoring, and human in the loop oversight to ensure reliability in a regulated healthcare setting. You will establish patterns, best practices, and tooling that enable scalable AI automation across teams.

Strong Pluses

• Experience with dbt or similar data transformation frameworks
• Familiarity with orchestration tools such as Airflow or Databricks Workflows
• Experience with agent evaluation and observability tools like LangSmith, Langfuse, or similar
• Background in healthcare, fintech, or other regulated high stakes domains where AI reliability matters
• Experience building internal developer tooling, platform capabilities, or developer facing products
• Hands on experience with retrieval augmented generation or grounding techniques for LLMs
• Familiarity with healthcare data formats and standards such as FHIR, HL7, claims data, and clinical NLP
• Experience with model evaluation, fairness assessment, or bias detection in ML/AI systems
• Understanding of healthcare regulations such as HIPAA and FDA guidance on AI/ML and responsible AI practices
• Experience establishing QA frameworks, test plans, and quality metrics for ML/AI systems
• Startup or high growth environment experience with rapid iteration and learning
• Published research, open source contributions, or demonstrated thought leadership in AI and agentic systems