Machine Learning Engineer (Real-Time Speech Translation)

LILT is building a live translation product, and is seeking an ML Engineer to own the real-time speech translation backend end-to-end, from live audio ingestion to translated output. The role focuses on integrating streaming ASR and adaptive MT into a low-latency system running on GPU Kubernetes infrastructure (Ray Serve).

Role Focus

• Own the backend that converts live audio into real-time translation results.
• Design for low latency by integrating streaming speech recognition and machine translation within defined latency budgets.
• Operate production services on GPU-accelerated Kubernetes, using Ray Serve.

Responsibilities

• Build and manage services for high-throughput, real-time audio and text streaming.
• Handle signal processing, session lifecycles, and concurrency management to ensure robust behavior under load.
• Integrate and serve streaming speech recognition and machine translation models in partnership with research teams to meet latency requirements.
• Develop logic for model-based confidence scoring, route segments for human intervention when needed, and broadcast real-time updates and corrections.
• Architect and scale production ML infrastructure on GPU-accelerated Kubernetes clusters.
• Implement batching, load balancing, and autoscaling strategies to maintain performance and cost efficiency.
• Establish comprehensive instrumentation for real-time performance monitoring.
• Identify bottlenecks, optimize system throughput, and reduce end-to-end latency to production standards.
• Define technical contracts and interfaces for audio ingestion and downstream service integrations.
• Partner with frontend and platform engineering teams to keep integration points clean and reliable.
• Drive cross-team alignment by defining API interfaces and technical contracts between engineering and product stakeholders.

Required Qualifications

• BS or MS in Computer Science or a related field, or equivalent practical experience.
• 3+ years building production backend or ML serving systems in Python, with strong async programming skills (asyncio).
• Hands-on experience with real-time streaming transport, including WebSocket or gRPC bidirectional streaming, session state, backpressure, and connection lifecycle handling.
• Experience serving ML models in production on GPUs using Ray Serve, Triton, vLLM, or similar, with Docker and Kubernetes.
• Experience integrating speech or NLP models into production systems, ideally streaming ASR (partial hypotheses, endpointing, VAD).
• A latency-engineering mindset, including profiling, instrumentation, and optimization of real-time or low-latency systems using per-stage budgets.
• Effective use of AI coding agents (Claude Code, Codex, or similar) with strong debugging, review, and reasoning skills.
• US citizenship and residence in the United States (contract requirement).

Preferred Qualifications

• Ray Serve experience, including streaming responses and model multiplexing.
• Familiarity with incremental or simultaneous machine translation concepts (retranslation, prefix stability, wait-k policies).
• Experience with machine translation quality estimation models (COMET/CometKiwi class) or other production confidence estimation approaches.
• Experience with message brokers for real-time fan-out and state distribution (RabbitMQ or similar).
• Streaming text-to-speech integration and time-to-first-audio optimization.
• WebRTC and SFU concepts, or familiarity with voice pipeline frameworks (LiveKit Agents, Pipecat).
• Experience handling CJK and other non-Latin text in NLP pipelines (Japanese, Korean, and English).
• Observability tooling experience for production ML systems (Datadog, Prometheus).

Technologies

• Python, asyncio
• WebSocket, gRPC
• Ray Serve, Triton, vLLM
• Docker, Kubernetes
• Claude Code, Codex
• COMET, CometKiwi
• RabbitMQ
• Datadog, Prometheus
• WebRTC, SFU, LiveKit Agents, Pipecat

Location and Eligibility

• Location: Boston, MA (onsite)
• Eligibility: US citizenship and residence in the United States
• Preferred locations: Washington, D.C.; Boston, MA; Indianapolis, IN (East Coast / ET timezone preferred)

Compensation

USD 120,000 - 161,434 per year.