Sr AI Engineer II

American Express is seeking a Sr AI Engineer II to help design and build foundational agentic AI platform capabilities for the enterprise (onsite in Sunrise, FL).

Responsibilities

• Contribute to the architecture and implementation of Agentic AI Platform capabilities, including Agent Runtime & Execution
• Design and build scalable agent runtime infrastructure using Kubernetes and cloud-native technologies
• Develop runtime abstractions enabling agents to run in internally managed sandboxed environments and third-party AI platforms
• Build core orchestration capabilities: tool execution, state and context management, memory, asynchronous workloads, event-driven execution, and multi-agent workflows
• Create a unified control plane to manage agents across heterogeneous execution environments and AI providers
• Develop APIs and services for agent registration, configuration, deployment, versioning, lifecycle management, policy enforcement, and runtime management
• Create abstractions that limit provider-specific complexity while preserving access to differentiated capabilities across AI ecosystems
• Engineer enterprise AI governance controls covering agent identity, authentication and authorization, tool permissions, policy enforcement, data boundaries, auditability, and lifecycle controls
• Build governance mechanisms for models, prompts, tools, MCP servers, knowledge sources, agent-to-agent interactions, and external integrations
• Partner with security, risk, privacy, and governance teams to translate enterprise requirements into scalable technical controls
• Build registries and catalogs for developer and AI system discovery of reusable agents, tools, skills, prompts, models, knowledge sources, and other platform capabilities
• Implement metadata and ecosystem health mechanisms: ownership, versioning, dependency management, certification, and discovery
• Deliver end-to-end telemetry for agent execution: traces, events, model interactions, tool calls, latency, token usage, cost, failures, policy decisions, and quality signals
• Make multi-agent and complex workflows explainable and debuggable to support platform and application teams
• Develop evaluation frameworks for quality, reliability, safety, and task performance
• Build automated evaluation pipelines using offline evaluations, production signals, human feedback, regression testing, and experimentation
• Establish continuous learning loops that turn production telemetry and feedback into improvements to agents, prompts, tools, models, and platform capabilities
• Create developer enablement via SDKs, APIs, CLIs, templates, local development environments, and self-service workflows
• Build paved roads that bake in enterprise security, governance, observability, and operational standards by default
• Develop AI-agent CI/CD: automated evaluation, policy validation, security checks, artifact/version management, deployment, promotion, rollback, and release controls
• Design GitOps and infrastructure-as-code patterns for deploying and managing agent workloads
• Help establish engineering standards for moving agents from experimentation to production safely and repeatedly

Requirements

• Strong software engineering experience building production systems using Python, Java, Go, or TypeScript
• Strong understanding of Generative AI, LLMs, agent architectures, tool/function calling, retrieval, context management, and agent orchestration
• Experience building production applications or platforms using major model providers or AI platforms
• Experience with Kubernetes, containers, microservices, distributed systems, and cloud-native architecture
• Experience designing production APIs and event-driven or asynchronous systems
• Strong understanding of modern cloud infrastructure and infrastructure-as-code practices
• Experience with CI/CD, automated testing, production observability, and software delivery practices
• Strong understanding of security fundamentals: identity, authentication, authorization, secrets, and least-privilege access
• Ability to navigate ambiguous technical problems and turn emerging technologies into reliable production systems
• Strong communication skills and ability to collaborate across engineering, architecture, product, security, and governance organizations
• Bachelor’s degree required

Technologies

• Python, Java, Go, TypeScript
• Kubernetes, cloud-native technologies
• CI/CD, GitOps, infrastructure-as-code
• OpenTelemetry
• OpenAI Agents SDK, LangGraph, Semantic Kernel, Google ADK
• Model Context Protocol (MCP)

Focus Areas (from the role scope)

• Agent Runtime & Execution: Kubernetes-based runtime infrastructure, sandboxed execution, orchestration, tools, state/context, memory, async and event-driven execution, multi-agent workflows
• Enterprise Agent Control Plane: unified management across heterogeneous environments and providers; agent registration, configuration, deployment, versioning, lifecycle, policy enforcement, runtime management; provider abstraction
• Agent Governance & Security: identity, authentication/authorization, tool permissions, policy enforcement, data boundaries, auditability, governance for models/prompts/tools/MCP/knowledge sources and integrations; collaboration with governance stakeholders
• Agent Discovery & Ecosystem: registries/catalogs, metadata, ownership, versioning, dependencies, certification, and discovery for a healthy enterprise ecosystem
• Agent Observability: telemetry for traces/events/model interactions/tool calls/latency/token usage/cost/failures/policy decisions/quality signals; explainability and debuggability; cross-model/runtime/tool visibility
• Evaluation & Continuous Learning: evaluation frameworks and automated pipelines (offline + production signals + human feedback + regression + experimentation); feedback-to-improvement loops for agents and platform capabilities
• Developer Experience: SDKs/APIs/CLIs/templates, local development environments, paved roads with embedded enterprise standards
• AI CI/CD & Platform Engineering: AI-agent CI/CD with evaluation, policy and security validation, artifact/version management, deployment/promotion/rollback/release controls; GitOps and infrastructure-as-code patterns; standards for safe experimentation to production

Preferred Qualifications

• Agent frameworks and orchestration technologies such as LangGraph, Semantic Kernel, Google ADK, OpenAI Agents SDK, or similar frameworks
• Model Context Protocol (MCP) and emerging agent interoperability protocols like A2A
• Multi-agent systems and agent-to-agent communication patterns
• Kubernetes operators, controllers, service meshes, or sophisticated Kubernetes platform engineering
• Agent/model gateways and intelligent model routing
• Vector databases, retrieval systems, embeddings, and enterprise knowledge architectures
• LLM and agent evaluation frameworks, LLM-as-judge techniques, human feedback systems, experimentation, and quality measurement
• AI observability, distributed tracing, OpenTelemetry, Langfuse and production monitoring
• Experience with Evals
• Policy engines and policy-as-code
• AI security: prompt injection defenses, tool security, data-loss prevention, AI-specific threat modeling
• Platform engineering: internal developer platforms, developer portals, enterprise service catalogs
• Large-scale distributed systems operating in highly regulated environments

Compensation: USD 123,000 - 215,250 per year.