Mountain View onsite role at ServiceNow building runtime infrastructure for Moveworks’ AI agents. You’ll work on the systems behind agent orchestration, distributed session management, event-driven pipelines, and production-grade observability, with an emphasis on engineering reliability at scale.
What you’ll build
As a Staff Software Engineer, you will help deliver a distributed execution platform for long-running agent sessions. The system coordinates multiple LLM calls and tool invocations, maintains session ownership, streams ordered messages, and provides traceable operations from the first request through minute-long workflows.
Responsibilities
- Agent orchestration engine: implement a state machine that manages long-running agent sessions, coordinating planning, execution, and user interaction across multiple LLM calls and tool invocations
- Distributed session management: build lease-based ownership using DynamoDB conditional writes, maintain liveness with heartbeat protocols, and recover from failures using checkpointing
- Event-driven message pipeline: use SQS FIFO queues for ordered delivery, Kafka consumers for event processing, and enable real-time streaming via gRPC and Socket.IO
- Structured concurrency: run concurrent session tasks with Python asyncio TaskGroups including message polling, lease heartbeats, output publishing, and orchestrator execution, using fail-fast semantics with graceful cancellation
- Observability infrastructure: instrument with OpenTelemetry, propagate distributed trace context across async boundaries, and manage custom span lifecycles for sessions that span minutes
- Caching and state layers: operate Redis and DynamoDB KV stores with per-org/per-bot scoping, optimize batch reads, and support hot-reload configuration
Requirements
- Deep experience in at least 3 of the following: Distributed systems (consistency models, idempotency, exactly-once delivery, distributed locking/leasing)
- Deep experience in at least 3 of the following: Concurrent/async programming (Python asyncio, Go goroutines, structured concurrency, cancellation handling)
- Deep experience in at least 3 of the following: Event-driven architectures (SQS, Kafka, pub/sub, backpressure, delivery guarantees)
- Deep experience in at least 3 of the following: Database systems for infrastructure (DynamoDB conditional writes, transactions; Redis connection pooling, pub/sub)
- Deep experience in at least 3 of the following: Observability (OpenTelemetry, distributed tracing, span context propagation, Prometheus metrics)
- Deep experience in at least 3 of the following: gRPC/protobuf (streaming RPCs, service interface design, error handling patterns)
- Strong in Python or Go (ideally both)
- Experience designing and operating systems that handle real traffic at scale
- Comfort with ambiguity for novel problems without textbook solutions
Technologies
Python, Go, DynamoDB, SQS FIFO, Kafka, gRPC, Socket.IO, Python asyncio TaskGroups, OpenTelemetry, Redis, Prometheus, gRPC/protobuf, LLM calls, proprietary models
Work persona
Work personas (flexible, remote, or required in office) are assigned based on the nature of the work and assigned work location. ServiceNow may confirm the distance between your primary residence and the closest ServiceNow office using a third-party service.
Equal opportunity employer
ServiceNow is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, creed, religion, sex, sexual orientation, national origin or nationality, ancestry, age, disability, gender identity or expression, marital status, veteran status, or any other category protected by law. Applicants with arrest or conviction records will be considered in accordance with legal requirements.
Accommodations
If you require a reasonable accommodation to complete any part of the application process, or are unable to use this online application and need an alternative method to apply, contact globaltalentss@servicenow.com for assistance.
Export control regulations
For positions requiring access to controlled technology subject to export control regulations, ServiceNow may be required to obtain export control approval from government authorities for certain individuals. Employment is contingent upon ServiceNow obtaining any export license or other approval required by relevant export control authorities.
