Llm Evaluation Engineer Jobs

1 job found

Today

Lead AI Engineer

Capital One New York, NY

New York, NY

Today

$215k - $246k/yr Full time Ai Governance Ai Ml Ai Observability

Apply Now

Job Description

Lead AI Engineer at Capital One (New York, NY) focuses on responsible, reliable AI systems across training, inference, evaluation, governance, and observability.

Responsibilities

Collaborate with cross-functional partners including engineers, research scientists, technical program managers, and product managers to deliver AI-powered products for associates and customers.
Design, develop, test, deploy, and support AI software components such as foundation model training, large language model inference, similarity search, guardrails, model evaluation, experimentation, governance, and observability.
Apply Open Source and SaaS AI technologies including AWS Ultraclusters, Huggingface, VectorDBs, Nemo Guardrails, PyTorch, and related tooling.
Develop and introduce state-of-the-art LLM optimization methods to improve production performance including scalability, cost, latency, and throughput.
Help shape the technical vision and long-term roadmap for foundational AI systems at Capital One.

Requirements

Bachelor’s or Master’s degree in Computer Science, AI, Electrical Engineering, Computer Engineering, or a related field with the following experience:
- Bachelor’s + at least 4 years developing AI and ML algorithms or technologies, or
- Master’s + at least 2 years developing AI and ML algorithms or technologies.
At least 4 years programming with Python, Go, Scala, or Java.

Preferred Qualifications

6 years deploying scalable and responsible AI solutions on cloud platforms such as AWS, Google Cloud, Azure, or equivalent private cloud.
Experience designing, developing, delivering, and supporting AI services.
Experience developing AI/ML algorithms or technologies (e.g., LLM inference, similarity search and VectorDBs, guardrails, memory) using Python, C++, C#, Java, or Golang.
Experience applying state-of-the-art training and inference optimization techniques to improve hardware utilization, latency, throughput, and cost.
Strong interest in staying current with AI research and applying novel techniques judiciously in production.

Core Technologies

AWS Ultraclusters, Huggingface, VectorDBs, Nemo Guardrails, PyTorch
Foundation model training, large language model inference, similarity search, guardrails
Model evaluation, experimentation, governance, observability
Python, Go, Scala, Java, AWS, Google Cloud, Azure, C++, C#, Golang

Team

The Intelligent Foundations and Experiences (IFX) team brings Capital One’s AI vision to life.
IFX partners across the company to advance science and AI engineering and build and deploy proprietary solutions central to business value for millions of customers.
AI models and platforms help teams enhance products with responsible, scalable AI for high-leverage impact.

Benefits

Comprehensive, competitive, and inclusive health, financial, and other benefits supporting total well-being.
Performance-based incentive compensation, which may include cash bonus(es) and/or long-term incentives (LTI).

Additional Information

Capital One may sponsor new qualified applicants for employment authorization.
Expected application review window: minimum of 5 business days.
No agencies please.
Equal opportunity employer (EOE, including disability/vet) with non-discrimination in compliance with applicable laws.
Promotes a drug-free workplace.
May consider qualified applicants with a criminal history consistent with applicable laws.
Accommodation available via Capital One Recruiting: 1-800-304-9102 or RecruitingAccommodation@capitalone.com.
Recruiting process questions: Careers@capitalone.com.
Capital One does not provide, endorse, or guarantee third-party products, services, educational tools, or other information available through this site.
Capital One Financial includes multiple entities; postings for Canada, the United Kingdom, and the Philippines correspond to specific entities.

Salary

New York, NY: USD 215,200 - 245,600 per year
Candidates hired in other locations will be subject to the pay range associated with that location.
Other listed ranges (full-time annual): Cambridge, MA: $197,300 - $225,100; McLean, VA: $197,300 - $225,100; San Francisco, CA: $215,200 - $245,600; San Jose, CA: $215,200 - $245,600.

View full job page →

Get Llm Evaluation Engineer Jobs alerts

New Llm Evaluation Engineer Jobs, daily.