Machine Learning Inference Engineer Jobs

1 job found

Today

AI Engineer 5 (MLX, Agentic AI, Gen AI platform Services)

Capital One San Jose, CA

San Jose, CA

Today

$251k - $286k/yr Full time Agentic Ai Artificial Intelligence Generative AI

Apply Now

Job Description

Capital One is hiring an AI Engineer 5 to build and support AI-powered products and foundational AI systems in an onsite role in San Jose, CA.

Responsibilities

Collaborate with cross-functional partners including engineers, research scientists, technical program managers, and product managers to deliver AI-powered products
Design, develop, test, deploy, and support AI software components, including foundation model training, LLM inference, agents and multi-agent workflows, similarity search, guardrails, model evaluation, experimentation, governance, and observability
Apply Open Source and SaaS AI tooling such as AWS Ultraclusters, Huggingface, VectorDBs, and PyTorch
Develop and introduce foundation model optimization techniques to improve scalability, cost, latency, and throughput for large-scale production AI
Contribute to the technical vision and long-term roadmap for foundational AI systems at Capital One
Design and optimize multi-model orchestration pipelines integrating LLMs, vector search, and domain-specific models
Establish and lead cost-performance governance reviews across AI systems, including GPU utilization, model throughput, and inference cost efficiency
Lead team design councils or design review boards to support technical consistency and compliance with AI engineering standards
Mentor Principal and Manager-level AI engineers to support cross-domain learning and raise organizational technical maturity

Requirements

Education + experience: Bachelor’s degree in Computer Science, AI, Electrical Engineering, Computer Engineering, or related field plus at least 6 years experience developing AI/ML algorithms or technologies; or Master’s degree plus at least 4 years experience
Programming: at least 6 years programming with Python, Go, Scala, CUDA, or Java

Technologies

AWS Ultraclusters
Huggingface
VectorDBs
PyTorch
AWS, Google Cloud, Azure
Python, Go, Scala, CUDA, Java, C++, C#, Golang
LLMs, vector search, GPU utilization
Rule-based, retrieval-augmented, generative components

Benefits

Comprehensive, competitive, and inclusive set of health, financial, and other benefits supporting total well-being
Performance-based incentive compensation, which may include cash bonus(es) and/or long term incentives (LTI)

Team

Intelligent Foundations and Experiences (IFX) team brings Capital One’s AI vision to life
Works with partners across the company to advance state-of-the-art science and AI engineering
Builds and deploys proprietary solutions central to the business, delivering value to millions of customers
Enables teams across Capital One to enhance products with responsible, scalable AI

Ideal Candidate

Enjoys building systems, taking pride in quality work, and doing the right thing
Interested in problems that help change banking for good
Stays current with AI research, can interpret scientific publications, and applies novel techniques in production
Thrives on bringing clarity to big, undefined problems
Asks questions, digs to find root causes, and communicates findings clearly
Shares new ideas even when they are unproven
Deeply technical with strong engineering and mathematics foundations
Understands optimization opportunities across hardware, software, and AI
Resilient trailblazer able to forge new paths toward business goals when the route is unknown

Preferred Qualifications

Experience leading development AI systems with tradeoff decisions across cost, latency, throughput, and accuracy
7 years of experience deploying scalable and responsible AI solutions on cloud platforms (AWS, Google Cloud, Azure, or equivalent private cloud)
Experience designing, developing, delivering, and supporting complex AI systems
Experience developing AI/ML algorithms such as LLM inference, similarity search and VectorDBs, guardrails, and memory using Python, C++, C#, Java, CUDA, or Golang
Experience optimizing training and inference software for hardware utilization, latency, throughput, and cost
Experience building agentic AI systems and agentic workflows
Experience architecting and integrating heterogeneous AI systems including rule-based, retrieval-augmented, and generative components into unified production pipelines
Experience defining and enforcing ethical AI deployment standards (explainability, fairness, and human-in-the-loop review processes)
Ability to balance model performance and operational cost using dynamic inference and model compression
Experience right-sizing models, instance counts, and hardware types based on requirements such as context length and token inputs/outputs

Salary and Location

Location: San Jose, CA (onsite)
Salary: USD 250,800 - 286,200 per year

View full job page →

Get Machine Learning Inference Engineer Jobs alerts

New Machine Learning Inference Engineer Jobs, daily.