Workflow Engines Engineer Jobs

1 job found

Today

Machine Learning Engineer, Agentic Systems

ServiceNow Mountain View, CA

Mountain View, CA

Today

Full time Backend Developer Agent Orchestration Ai Workflows

Apply Now

Job Description

This distributed systems engineering role builds the runtime infrastructure powering Moveworks AI agents in Mountain View, CA (onsite), focusing on orchestration, execution, and real-time delivery of agent responses rather than a traditional ML role.

Responsibilities

Agent orchestration engine: design and operate a state machine that manages long running agent sessions, coordinating planning, execution, and user interactions across multiple LLM calls and tool invocations
Distributed session management: enforce lease based ownership through DynamoDB conditional writes, maintain heartbeats, and enable crash recovery via checkpointing
Event driven messaging pipeline: leverage SQS FIFO for ordered delivery, process events with Kafka consumers, and stream in real time using gRPC and Socket.IO
Structured concurrency: use Python asyncio TaskGroups to run multiple concurrent tasks per session (polling messages, lease heartbeats, publishing outputs, orchestrator work) with fail fast semantics and clean cancellation
Observability: instrument with OpenTelemetry, propagate distributed trace context across async boundaries, and manage custom span lifecycles for minutes-long sessions
Caching and state management: Redis and DynamoDB key-value stores scoped per organization and per bot, optimize batch reads, and support hot reload of configuration

Requirements

Distributed systems: experience with consistency models, idempotency, exactly once delivery, and distributed locking or leasing
Concurrent and async programming: strong skills in Python asyncio or Go goroutines, with structured concurrency and cancellation handling
Event driven architectures: familiarity with message queues (SQS, Kafka), pub/sub patterns, backpressure, and delivery guarantees
Database systems for infrastructure: DynamoDB (conditional writes, transactions) and Redis (connection pooling, pub/sub)
Observability: OpenTelemetry, distributed tracing, span context propagation, and Prometheus metrics
gRPC and protobuf: streaming RPCs, service interface design, and common error handling patterns
2+ years building production backend or infrastructure systems
Strong in Python or Go, ideally both
Experience designing and operating systems that handle real traffic at scale
Comfort with ambiguity and solving novel problems without standard solutions

Technologies

Python
Go
asyncio
DynamoDB
Redis
SQS
Kafka
gRPC
protobuf
Socket.IO
OpenTelemetry
Prometheus

View full job page →

Get Workflow Engines Engineer Jobs alerts

New Workflow Engines Engineer Jobs, daily.