Machine Learning Engineer 5, Senior Manager - Intelligent Foundations and Experiences (IFX)

This role is for a Senior Manager focused on building and scaling AI and machine learning capabilities, including model training and serving in production and multi-tenant ML platforms. You will work with Product and Data Science teams to deliver reliable data pipelines, CI/CD workflows, and Responsible and Explainable AI practices.

Location

McLean, VA (onsite)

Salary

• McLean, VA: $229,900 - $262,400 per year (Machine Learning Engineer 5)
• Richmond, VA: $209,000 - $238,500 per year (Machine Learning Engineer 5)

Role Summary

You will design and deliver ML models and platform components that address business needs, while using your depth in modeling and ML infrastructure decisions. The work includes training, deploying, and maintaining models in production, scaling large footprint training and serving, and ensuring governance and security best practices for risk and Responsible and Explainable AI.

Responsibilities

• Design, build, and/or deliver ML models and components to solve real-world business problems in collaboration with Product and Data Science teams
• Build and scale multi-tenant platforms enabling large footprint ML training and/or serving at scale
• Use ML modeling fundamentals to guide infrastructure decisions, including model choice, data and feature selection, training, hyperparameter tuning, dimensionality, bias/variance, and validation
• Solve complex problems by writing and testing application code, developing and validating ML models, and automating tests and deployment
• Collaborate in a cross-functional Agile team to create and enhance software for big data and ML applications
• Retrain, maintain, and monitor models in production
• Leverage or build cloud-based architectures, technologies, and/or platforms to deliver optimized ML models at scale
• Construct optimized data pipelines to feed ML models
• Apply CI/CD best practices, including test automation and monitoring, to support successful model and application deployments
• Ensure code is well-managed to reduce vulnerabilities, models are well-governed from a risk perspective, and ML follows Responsible and Explainable AI best practices
• Use programming languages such as Python, Scala, or Java

Required Qualifications

• Bachelor's Degree or higher in Computer Science, Machine Learning, or a related quantitative field (Statistics, Economics, Operations Research, Analytics, Mathematics, Engineering)
• At least 6 years of experience programming with Python, Java, Golang, or C++
• At least 6 years of Machine Learning experience using industry standard frameworks PyTorch or Tensorflow and libraries Pandas, NumPy, Scikit-learn
• At least 6 years of experience using and operating large scale distributed systems (Spark, Ray) to prepare AI or Machine Learning data
• At least 4 years of experience deploying and operating Machine Learning solutions in production and operating production services in the cloud (AWS, GCP, Azure) and using Kubernetes to manage large scale containerized Machine Learning systems

Technologies

• Python, Scala, Java, Golang, C++
• PyTorch, Tensorflow, Pandas, NumPy, Scikit-learn
• Spark, Ray
• AWS, GCP, Azure
• Kubernetes, CI/CD

Benefits

• Eligible for performance based incentive compensation, which may include cash bonus(es) and/or long term incentives (LTI)
• Capital One offers a comprehensive, competitive, and inclusive set of health, financial and other benefits supporting total well-being

Preferred Qualifications

• Master's or Doctoral Degree in Computer Science, Electrical Engineering, Mathematics, or related field
• +2 years experience building production RAG pipelines (chunking, embeddings, vector indexing, graph RAG, or fine-tuning)
• +2 years experience leveraging Agentic patterns (Tool Calling, ReAct loops, Plan-and-Execute, Multi-Agent Collaboration)
• +2 years with LLM orchestration frameworks (LangChain, LangGraph, LlamaIndex, Semantic Kernel) and model APIs (OpenAI, Anthropic, Bedrock, Ollama, Hugging Face)
• +2 year experience in building agentic AI systems and agentic workflows
• +2 year experience developing AI and ML algorithms or technologies (e.g. LLM Inference, Similarity Search and VectorDBs, Guardrails, Memory) using Python, C++, C#, Java, CUDA, or Golang
• +2 years experience right-sizing models, instance counts, and hardware types given requirements (e.g., context length, token inputs, token outputs)
• 5+ years of experience optimizing ML algorithms, configurations, and infrastructure
• 5+ years of experience following software development best practices including source control, testing, code reviews, CI/CD, etc.
• 5+ years of experience building resilient software solutions with pre-production testing, advanced deployment techniques (one-box, blue/green, gradual dial-up), monitoring, alarms, and preparing incident response plans
• 5+ years of experience working with Machine Learning techniques and model types, including evaluation and diagnosing issues such as underfitting and overfitting
• 5+ years of experience designing, implementing, and scaling production-ready data pipelines for training and evaluating ML models
• ML industry impact through conference presentations, papers, blog posts, open source contributions, or patents
• Ability to communicate complex technical and machine learning concepts clearly to a variety of audiences