Open Observability Engineer Jobs

1 job found

Today

Sr AI Engineer II

American Express Sunrise, FL

Sunrise, FL

Today

$123k - $215k/yr Full time Ai Agent Ai Agent Platform Ai Orchestration

Apply Now

Job Description

American Express is seeking a Sr AI Engineer II to help design and build foundational agentic AI platform capabilities for the enterprise (onsite in Sunrise, FL).

Responsibilities

Contribute to the architecture and implementation of Agentic AI Platform capabilities, including Agent Runtime & Execution
Design and build scalable agent runtime infrastructure using Kubernetes and cloud-native technologies
Develop runtime abstractions enabling agents to run in internally managed sandboxed environments and third-party AI platforms
Build core orchestration capabilities: tool execution, state and context management, memory, asynchronous workloads, event-driven execution, and multi-agent workflows
Create a unified control plane to manage agents across heterogeneous execution environments and AI providers
Develop APIs and services for agent registration, configuration, deployment, versioning, lifecycle management, policy enforcement, and runtime management
Create abstractions that limit provider-specific complexity while preserving access to differentiated capabilities across AI ecosystems
Engineer enterprise AI governance controls covering agent identity, authentication and authorization, tool permissions, policy enforcement, data boundaries, auditability, and lifecycle controls
Build governance mechanisms for models, prompts, tools, MCP servers, knowledge sources, agent-to-agent interactions, and external integrations
Partner with security, risk, privacy, and governance teams to translate enterprise requirements into scalable technical controls
Build registries and catalogs for developer and AI system discovery of reusable agents, tools, skills, prompts, models, knowledge sources, and other platform capabilities
Implement metadata and ecosystem health mechanisms: ownership, versioning, dependency management, certification, and discovery
Deliver end-to-end telemetry for agent execution: traces, events, model interactions, tool calls, latency, token usage, cost, failures, policy decisions, and quality signals
Make multi-agent and complex workflows explainable and debuggable to support platform and application teams
Develop evaluation frameworks for quality, reliability, safety, and task performance
Build automated evaluation pipelines using offline evaluations, production signals, human feedback, regression testing, and experimentation
Establish continuous learning loops that turn production telemetry and feedback into improvements to agents, prompts, tools, models, and platform capabilities
Create developer enablement via SDKs, APIs, CLIs, templates, local development environments, and self-service workflows
Build paved roads that bake in enterprise security, governance, observability, and operational standards by default
Develop AI-agent CI/CD: automated evaluation, policy validation, security checks, artifact/version management, deployment, promotion, rollback, and release controls
Design GitOps and infrastructure-as-code patterns for deploying and managing agent workloads
Help establish engineering standards for moving agents from experimentation to production safely and repeatedly

Requirements

Strong software engineering experience building production systems using Python, Java, Go, or TypeScript
Strong understanding of Generative AI, LLMs, agent architectures, tool/function calling, retrieval, context management, and agent orchestration
Experience building production applications or platforms using major model providers or AI platforms
Experience with Kubernetes, containers, microservices, distributed systems, and cloud-native architecture
Experience designing production APIs and event-driven or asynchronous systems
Strong understanding of modern cloud infrastructure and infrastructure-as-code practices
Experience with CI/CD, automated testing, production observability, and software delivery practices
Strong understanding of security fundamentals: identity, authentication, authorization, secrets, and least-privilege access
Ability to navigate ambiguous technical problems and turn emerging technologies into reliable production systems
Strong communication skills and ability to collaborate across engineering, architecture, product, security, and governance organizations
Bachelor’s degree required

Technologies

Python, Java, Go, TypeScript
Kubernetes, cloud-native technologies
CI/CD, GitOps, infrastructure-as-code
OpenTelemetry
OpenAI Agents SDK, LangGraph, Semantic Kernel, Google ADK
Model Context Protocol (MCP)

Focus Areas (from the role scope)

Agent Runtime & Execution: Kubernetes-based runtime infrastructure, sandboxed execution, orchestration, tools, state/context, memory, async and event-driven execution, multi-agent workflows
Enterprise Agent Control Plane: unified management across heterogeneous environments and providers; agent registration, configuration, deployment, versioning, lifecycle, policy enforcement, runtime management; provider abstraction
Agent Governance & Security: identity, authentication/authorization, tool permissions, policy enforcement, data boundaries, auditability, governance for models/prompts/tools/MCP/knowledge sources and integrations; collaboration with governance stakeholders
Agent Discovery & Ecosystem: registries/catalogs, metadata, ownership, versioning, dependencies, certification, and discovery for a healthy enterprise ecosystem
Agent Observability: telemetry for traces/events/model interactions/tool calls/latency/token usage/cost/failures/policy decisions/quality signals; explainability and debuggability; cross-model/runtime/tool visibility
Evaluation & Continuous Learning: evaluation frameworks and automated pipelines (offline + production signals + human feedback + regression + experimentation); feedback-to-improvement loops for agents and platform capabilities
Developer Experience: SDKs/APIs/CLIs/templates, local development environments, paved roads with embedded enterprise standards
AI CI/CD & Platform Engineering: AI-agent CI/CD with evaluation, policy and security validation, artifact/version management, deployment/promotion/rollback/release controls; GitOps and infrastructure-as-code patterns; standards for safe experimentation to production

Preferred Qualifications

Agent frameworks and orchestration technologies such as LangGraph, Semantic Kernel, Google ADK, OpenAI Agents SDK, or similar frameworks
Model Context Protocol (MCP) and emerging agent interoperability protocols like A2A
Multi-agent systems and agent-to-agent communication patterns
Kubernetes operators, controllers, service meshes, or sophisticated Kubernetes platform engineering
Agent/model gateways and intelligent model routing
Vector databases, retrieval systems, embeddings, and enterprise knowledge architectures
LLM and agent evaluation frameworks, LLM-as-judge techniques, human feedback systems, experimentation, and quality measurement
AI observability, distributed tracing, OpenTelemetry, Langfuse and production monitoring
Experience with Evals
Policy engines and policy-as-code
AI security: prompt injection defenses, tool security, data-loss prevention, AI-specific threat modeling
Platform engineering: internal developer platforms, developer portals, enterprise service catalogs
Large-scale distributed systems operating in highly regulated environments

Compensation: USD 123,000 - 215,250 per year.

View full job page →

Get Open Observability Engineer Jobs alerts

New Open Observability Engineer Jobs, daily.