Gpu Inference

Gpu Inference Engineer Jobs

3 jobs found

1d ago

Senior Machine Learning Engineer

Cloudflare New York, NY

New York, NY

1d ago

$194k - $266k/yr Full time Artificial Intelligence Generative AI Gpu Inference

Apply Now

Job Description

Cloudflare is building machine learning inference that can run across its global network, supporting everything from frontier open LLMs and real-time voice models to customer-deployed models served on heterogeneous GPUs and next-generation accelerators. In this role, you will help define how models operate at scale, with an emphasis on low latency, reliability, and efficient resource use.

Working on Cloudflare’s serverless inference platform, you will develop production-grade ML inference and the evaluation tooling that measures model quality and serving performance across multiple model families, including LLMs, speech, and vision.

Responsibilities

Develop, optimize, and productionize machine learning models for Cloudflare’s serverless inference platform, focusing on performance, reliability, and model quality.
Build benchmarking and evaluation frameworks to measure latency, throughput, cost efficiency, and model behavior across LLMs, speech, vision, and other model families.
Improve inference performance using quantization, batching, caching, model compilation, runtime tuning, and accelerator-aware optimization.
Collaborate with systems engineers to integrate models into Cloudflare’s distributed inference infrastructure across a heterogeneous fleet of GPUs and next-generation accelerators.
Advance model deployment workflows, including validation, rollout safety, observability, regression testing, and operational readiness.
Work with product and engineering teams to translate customer requirements into scalable ML capabilities for Workers AI.
Mentor engineers, contribute to technical direction, and raise the quality bar for production ML engineering practices across the team.

Requirements

Experience building, optimizing, and operating machine learning models in production environments.
Strong proficiency with Python and modern ML frameworks such as PyTorch, TensorFlow, JAX, or equivalents.
Hands-on inference optimization experience for large-scale models, including quantization, batching, caching, compilation, and serving runtime tuning.
Experience with large-scale inference serving frameworks or runtimes such as SGLang, vLLM, TensorRT-LLM, ONNX Runtime, Triton, llama.cpp, or similar.
Familiarity with LLMs, speech models, vision models, embeddings, multimodal models, retrieval-augmented generation, or other modern deep learning architectures.
Experience optimizing models for GPUs or specialized accelerators.
Strong understanding of production ML concerns, including evaluation, monitoring, model regressions, rollout safety, and reliability.
Ability to collaborate across ML and systems boundaries, including familiarity with distributed systems, networking, or serverless platforms.
Track record of leading complex technical projects and mentoring other engineers.

Technologies

Python
PyTorch
TensorFlow
JAX
SGLang
vLLM
TensorRT-LLM
ONNX Runtime
Triton
llama.cpp
Workers AI

Benefits

Medical/Rx Insurance
Dental Insurance
Vision Insurance
Flexible Spending Accounts
Commuter Spending Accounts
Fertility & Family Forming Benefits
On-demand mental health support and Employee Assistance Program
Global Travel Medical Insurance
Short and Long Term Disability Insurance
Life & Accident Insurance
401(k) Retirement Savings Plan
Employee Stock Participation Plan
Flexible paid time off covering vacation and sick leave
Leave programs, including parental, pregnancy health, medical, and bereavement leave

Bonus Points

Experience contributing to open source ML tooling, model serving frameworks, or inference runtimes.

Compensation

Compensation may be adjusted depending on work location.
For Bay Area based hires: Estimated annual salary of $194,000 - $266,000.
For New York City, New Jersey, Washington, Washington DC, and California (excluding Bay Area) based hires: Estimated annual salary of $185,000 - $254,000.

Equity

This role is eligible to participate in Cloudflare’s equity plan.

Location

New York, NY (hybrid)

View full job page →

13d ago

Software Engineer, DL Kernel Libraries - New College Grad 2027

NVIDIA Santa Clara, CA

Santa Clara, CA

13d ago

$108k - $196k/yr Internship Cuda Deep Learning Gpu Inference

Apply Now

Job Description

This internship role supports NVIDIA’s DL Kernel Libraries team by building software stack components for deep learning inference and training, spanning GPU kernel technologies, libraries, and code generation efforts. The work emphasizes practical performance and extensibility across modern AI workloads.

Role Overview

As a Software Engineer Intern in the DL Kernel Libraries group, you will contribute to inference and training systems software, including kernel implementations and compiler or runtime components. Focus areas include cuDNN, FlashInfer, efficient kernel engineering, and domain-specific tooling that accelerates AI workloads.

Key Responsibilities

Innovate and develop new AI systems technologies aimed at efficient inference and training
Design, implement, and optimize kernels for high-impact AI workloads
Design and implement extensible abstractions for serving engines and training frameworks
Build efficient just-in-time domain specific compilers and runtimes
Collaborate closely with engineers across NVIDIA spanning deep learning frameworks, libraries, kernels, and GPU architecture teams
Contribute to open source communities across FlashInfer, cuDNN, vLLM, SGLang, PyTorch, JAX, and XLA

Required Qualifications

BS, MS, or PhD degree in Computer Science, Electrical Engineering, or a related field, or equivalent experience
Strong experience in developing or using deep learning frameworks (such as PyTorch, JAX, TensorFlow, ONNX) or inference engines and runtimes such as vLLM, SGLang, and MLC
Strong Python and C/C++ programming skills

Technologies and Tools

PyTorch, JAX, TensorFlow, ONNX
vLLM, SGLang, MLC
Python, C/C++
FlashInfer, cuDNN
CUDA C/C++, CUTLASS DSL, cuTile, Triton
Flash Attention
Apache TVM, MLIR
CUDA, XLA

Salary and Location

Location: Santa Clara, CA (onsite)
Compensation: USD 108,000 to 195,500 per year
Job Type: Internship

Benefits

You will also be eligible for equity and benefits.

What Helps You Stand Out

Background in domain specific compiler and library solutions for LLM inference and training, such as FlashInfer and Flash Attention
Expertise in training frameworks and inference engines
Expertise in machine learning compilers, such as Apache TVM and MLIR
Strong experience in GPU kernel development and performance optimizations, including CUDA C/C++, CUTLASS DSL, cuTile, Triton, or similar approaches
Open source project ownership or contributions

View full job page →

1mo ago

Principal Software Engineer, Inference

Hewlett Packard Enterprise Spring, TX

Spring, TX

1mo ago Closed

$160k - $303k/yr Full time Backend Developer Artificial Intelligence Distributed Inference

This position is no longer accepting applications (closed Oct 18, 2026).

Job Description

Hewlett Packard Enterprise is building HPE AI Essentials, an inference platform designed to run large language models on customer-owned infrastructure, including air-gapped and sovereign environments. In this hybrid role in Spring, TX, you will help lead the model runtime and the Kubernetes orchestration approach that supports efficient, low-latency, high-utilization inference.

What you will do

Define and own the technical direction of the LLM serving deployment, including engine integration, continuous batching, KV cache management and reuse, and quantized execution.
Collaborate with inference performance engineering teams, with responsibility for time-to-first-token, inter-token latency, throughput per GPU, and P95/P99 tail latency.
Set the distributed inferencing strategy, including disaggregated prefill and decode, tensor and pipeline parallelism, and KV cache offload across GPU memory, host memory, and RDMA-attached storage.
Assess emerging runtimes and methods, including quantization schemes, speculative decoding, and mixture-of-experts serving, and determine whether to adopt, develop in-house, or decline each approach.
Define the orchestration layer for the runtime, including model admission, GPU scheduling and partitioning, cache-aware request routing, and autoscaling.
Mentor engineers, lead design and architecture reviews, and present technical direction to business unit and executive audiences.

Required qualifications

Production experience with LLM inference engines such as vLLM, SGLang, TensorRT-LLM, TGI, or NVIDIA NIM, including the ability to modify engine internals.
Deep understanding of inference internals, including continuous batching, paged attention, KV cache reuse and prefix caching, chunked prefill, quantization, and speculative decoding.
Knowledge of tensor and pipeline parallelism, NCCL collective operations, and how GPU memory hierarchy and interconnect characteristics affect performance.
Expert proficiency with Kubernetes platform architectures, including operators, custom resources, controllers, and scheduling.
Strong programming ability in Go and Python, plus the ability to read, debug, and profile C++/CUDA using tools such as Nsight.
Experience debugging and profiling multi-tier application workloads such as RAG and Agents.
Excellent analytical, debugging, and problem-solving skills.

Technologies involved

vLLM, SGLang, TensorRT-LLM, TGI, NVIDIA NIM
Go, Python, C++, CUDA, Nsight
Kubernetes, NCCL, RDMA, GPUDirect Storage, InfiniBand, RoCE
RAG, Agents

Education and experience

Minimum experience: 12 years
Education: Degree in Computer Science or a related field

Benefits

A comprehensive benefits suite intended to support physical, financial, and emotional wellbeing.
Programs designed to support career goals, whether you want to deepen expertise or apply your skills elsewhere within HPE.
Unconditional inclusion, with flexibility to manage work and personal needs.

Preferred background

Upstream contribution to vLLM, SGLang, TensorRT-LLM, LLM-D, LMCache, or KServe.
Experience with disaggregated prefill/decode serving, or KV cache offload and reuse at scale.
Experience with RDMA, GPUDirect Storage, InfiniBand, or RoCE.
Experience with MIG, fractional GPU allocation, and multi-tenant GPU isolation.
Experience delivering on-premises, air-gapped, or regulated enterprise software.

Location and work model

Location: Spring, TX (hybrid)
The role is designed as hybrid, with an expectation to work on average 2 days per week from an HPE office.
Primary work location is Spring, TX, with the possibility of other HPE sites in the US; remote work options may be considered.

Compensation

Annual base salary range: USD 160,000 - 303,000 (listed)
Annual salary range also varies by location: USD 160,000 - 303,000 in Colorado; USD 152,000 - 349,000 in North Carolina & Texas
Variable incentives may also be offered.

View full job page →

Gpu Inference Engineer Jobs

Senior Machine Learning Engineer

Software Engineer, DL Kernel Libraries - New College Grad 2027

Principal Software Engineer, Inference

Get Gpu Inference Engineer Jobs alerts