Machine Learning Engineer, Agentic Systems

This distributed systems engineering role builds the runtime infrastructure powering Moveworks AI agents in Mountain View, CA (onsite), focusing on orchestration, execution, and real-time delivery of agent responses rather than a traditional ML role.

Responsibilities

• Agent orchestration engine: design and operate a state machine that manages long running agent sessions, coordinating planning, execution, and user interactions across multiple LLM calls and tool invocations
• Distributed session management: enforce lease based ownership through DynamoDB conditional writes, maintain heartbeats, and enable crash recovery via checkpointing
• Event driven messaging pipeline: leverage SQS FIFO for ordered delivery, process events with Kafka consumers, and stream in real time using gRPC and Socket.IO
• Structured concurrency: use Python asyncio TaskGroups to run multiple concurrent tasks per session (polling messages, lease heartbeats, publishing outputs, orchestrator work) with fail fast semantics and clean cancellation
• Observability: instrument with OpenTelemetry, propagate distributed trace context across async boundaries, and manage custom span lifecycles for minutes-long sessions
• Caching and state management: Redis and DynamoDB key-value stores scoped per organization and per bot, optimize batch reads, and support hot reload of configuration

Requirements

• Distributed systems: experience with consistency models, idempotency, exactly once delivery, and distributed locking or leasing
• Concurrent and async programming: strong skills in Python asyncio or Go goroutines, with structured concurrency and cancellation handling
• Event driven architectures: familiarity with message queues (SQS, Kafka), pub/sub patterns, backpressure, and delivery guarantees
• Database systems for infrastructure: DynamoDB (conditional writes, transactions) and Redis (connection pooling, pub/sub)
• Observability: OpenTelemetry, distributed tracing, span context propagation, and Prometheus metrics
• gRPC and protobuf: streaming RPCs, service interface design, and common error handling patterns
• 2+ years building production backend or infrastructure systems
• Strong in Python or Go, ideally both
• Experience designing and operating systems that handle real traffic at scale
• Comfort with ambiguity and solving novel problems without standard solutions

Technologies

• Python
• Go
• asyncio
• DynamoDB
• Redis
• SQS
• Kafka
• gRPC
• protobuf
• Socket.IO
• OpenTelemetry
• Prometheus