EngineerJobs.io
← Back to all jobs
Gallatin

Principal AI Engineer

Washington, DC $240k - $310k/yr Full time Posted 2h ago

Job Description

Gallatin is building an AI team focused on AI-driven logistics planning and decision support for military operations. This onsite Principal AI Engineer role in Washington, DC serves as a technical anchor, setting architecture and quality standards while helping the team design, build, review, and ship production AI systems that integrate cleanly with an event-driven backend.

You will shape how AI services fit into the broader platform by owning scalability targets, integration patterns, and production code for difficult problems across agent and retrieval services, AI-specific backend capabilities, and data pipelines. Gallatin emphasizes engineering quality that scales, strong review culture, and practical delivery in regulated environments.

What you’ll do

  • Act as a quality gate for AI services, defining expectations for design, testing, and scalability, and reviewing the changes that matter most before release.
  • Own how AI services integrate with a NATS-based event-driven backend, including data models and APIs, working with platform and backend engineers to keep integrations safe, scalable, and consistent.
  • Provide trusted code review for AI changes, including updates to shared backend and data systems.
  • Build a scalable review culture through architecture guidelines, reusable design review templates, common patterns to follow and avoid, and risk-based review tiers, including AI-assisted review.
  • Prevent architectural drift by ensuring parallel work aligns to the same design, including when AI coding assistants are in use.
  • Set and enforce scalability targets for latency, throughput, concurrency, and cost, identify bottlenecks, and validate “what happens at 10x?” before shipping.
  • Write production code on complex challenges, including AI-specific backend features, agent and retrieval services, and data pipelines.
  • Lead design reviews and RFCs, driving decisions and guiding safe rollouts for infrastructure and data model changes.
  • Make build-versus-buy and model selection decisions, including self-hosted versus cloud tradeoffs under data-handling and deployment constraints.
  • Unblock the team’s progress by ensuring access to data sources, infrastructure, and tooling.
  • Mentor engineers and help grow future technical leads, including contributing to the hiring bar.

What you bring

  • 10+ years of software engineering experience, with demonstrated capability designing and scaling complex production systems across multiple orders of magnitude of growth.
  • Principal-level depth in backend and distributed systems, including service design, event-driven/messaging architectures, APIs, data modeling, concurrency, caching, and failure handling.
  • Proven track record as a code and architecture gatekeeper and experience spreading those practices across other engineers.
  • Strong computer science fundamentals applied with judgment.
  • Deep Python fluency and strong TypeScript (or comparable full-stack experience).
  • Production experience with, or rapid ramp capability on, LLM systems including RAG, agents, tool use, and evaluation.
  • Testing and observability instincts, including reliable test strategies for nondeterministic components.
  • Working fluency with cloud infrastructure, containers, and CI/CD, with evidence you can ship what you build.
  • Clear written and verbal communication with engineers, product leaders, and customers.
  • Eligibility for a US security clearance.

Technology focus

  • Python, TypeScript
  • NATS
  • LLM systems, RAG, agents, tool use, evaluation
  • Cloud infrastructure, containers, CI/CD

Bonus points

  • Prior principal or staff-level experience on a platform other teams build on.
  • Experience scaling engineering practices (review, design standards, onboarding) as headcount tripled.
  • Experience with data platforms, feature stores, or ML pipelines.
  • Experience with optimization, simulation, or logistics systems.
  • Experience with constrained, air-gapped, or regulated deployments.
  • Background in military logistics, planning doctrine, or defense programs.

Compensation range: $240,000 - $310,000 per year.

Similar Jobs