Ml Inference Engineer Jobs

1 job found

Today

AI Engineer 5 (Gen AI Platform Services: Agentic AI, Guardrails, Evaluation)

Capital One New York, NY

New York, NY

Today

$251k - $286k/yr Full time Backend Developer Agentic Ai Ai Agent

Apply Now

Job Description

The AI Engineer 5 role at Capital One focuses on building responsible and reliable GenAI platform services. The position contributes to AI software components that support foundation model training, LLM inference, agentic workflows, guardrails, and model evaluation using scalable cloud and open-source technologies.

Responsibilities

Collaborate with engineers, research scientists, technical program managers, and product managers to deliver AI-powered products that improve how associates work and how customers interact with Capital One.
Design, develop, test, deploy, and support AI software components, including foundation model training, large language model inference, agents and multi-agent workflows, similarity search, guardrails, and model evaluation, along with related capabilities such as experimentation, governance, and observability.
Use an engineering stack that includes Open Source and SaaS AI technologies, including AWS Ultraclusters, Huggingface, VectorDBs, and PyTorch.
Develop and introduce state-of-the-art foundation model optimization techniques to improve scalability, cost, latency, and throughput for large scale production AI systems.
Contribute to technical vision and long-term roadmap planning for foundational AI systems at Capital One.
Design and optimize multi-model orchestration pipelines by integrating LLMs, vector search, and domain-specific models into unified systems.
Lead cost-performance governance reviews across AI systems, including tracking GPU utilization, model throughput, and inference cost efficiency.
Participate in and lead team design councils or design review boards to support technical consistency and compliance with AI engineering standards.
Mentor Principal and Manager-level AI engineers to support cross-domain learning and increase organizational technical maturity.

Requirements

Bachelor’s degree in Computer Science, AI, Electrical Engineering, Computer Engineering, or a related field, plus at least 6 years of experience developing AI and ML algorithms or technologies; or a Master’s degree in the same or related fields, plus at least 4 years of experience developing AI and ML algorithms or technologies.
At least 6 years of programming experience with Python, Go, Scala, CUDA, or Java.

Technologies

AWS Ultraclusters, Huggingface, VectorDBs, PyTorch, AWS, Google Cloud, Azure
Python, Go, Scala, CUDA, Java, C++, C#, Golang
Rule-based, retrieval-augmented, generative components

Preferred Qualifications

Experience leading development AI systems with tradeoff decisions across cost, latency, throughput, and accuracy.
7+ years of experience deploying scalable and responsible AI solutions on cloud platforms such as AWS, Google Cloud, Azure, or equivalent private cloud.
Experience designing, developing, delivering, and supporting complex AI systems.
Experience building AI and ML algorithms or technologies (for example, LLM inference, similarity search and VectorDBs, guardrails, memory) using Python, C++, C#, Java, CUDA, or Golang.
Experience applying state-of-the-art techniques to optimize training and inference software for improved hardware utilization, latency, throughput, and cost.
Experience building agentic AI systems and agentic workflows.
Knowledge of recent AI research and the ability to apply novel techniques in production.
Excellent communication and presentation skills, including the ability to explain complex AI concepts to peers.
Experience architecting and integrating heterogeneous AI systems that combine rule-based, retrieval-augmented, and generative components into unified production pipelines.
Experience defining and enforcing standards for ethical AI deployment, including explainability, fairness, and human-in-the-loop review processes.
Ability to balance model performance with operational cost using dynamic inference strategies and model compression.
Experience right-sizing models, instance counts, and hardware types based on requirements such as context length and token inputs/outputs.

Benefits

Health, financial and other benefits that support your total well-being.
Performance based incentive compensation, which may include cash bonus(es) and/or long term incentives (LTI).
Comprehensive, competitive, and inclusive set of health, financial and other benefits.

Role Details

Location: New York, NY (onsite)
Salary: USD 250,800 - 286,200 per yearly
Minimum Experience: 6 years

View full job page →

Get Ml Inference Engineer Jobs alerts

New Ml Inference Engineer Jobs, daily.