Machine learning engineer II, AWS Inference Optimization

Machine learning engineer on an AWS inference optimization team focused on improving large language model inference performance.

Responsibilities

• Develop model servers for large language OSS models (examples include Qwen, Gemma, gptoss, and others)
• Build inference stack components, including CUDA kernel development in vLLM and SGLang
• Create and iterate on services for GenAI optimization and benchmarking
• Create and modify a mid-size application or service
• Apply inference optimization techniques to improve performance, including:
  • Speculative decoding
  • Disaggregated Inference
  • CUDA kernel tuning
• Own price-performance for Amazon LLM inference on SageMaker, Hyperpod, and Bedrock
• Deliver production-ready solutions for customer-facing products, using existing academic/industrial research and practical problem solving
• Optimize inference performance using AWS inference stacks including vLLM and SGLang
• Tune GenAI inference performance on Amazon SageMaker, Amazon Bedrock, and Hyperpod

Requirements

• 3+ years of non-internship professional software development experience
• 2+ years of non-internship experience designing or architecting new and existing systems (design patterns, reliability, scaling)
• 1+ years of software development engineer or related occupational experience
• 1+ years building large-scale, multi-tiered, multi-threaded, embedded, or distributed software applications, tools, systems, and services using C#, C++, Java, or Perl
• Bachelor’s degree (Computer Science, Engineering, Mathematics, or related field) or foreign equivalent
• Knowledge of Machine Learning and LLM fundamentals, including:
  • Transformer architecture
  • Training and inference lifecycles
  • Optimization techniques

Technologies

• vLLM, SGLang
• Amazon SageMaker, Amazon Bedrock, Hyperpod
• CUDA
• C#, C++, Java, Perl
• Transformer architecture

Preferred Qualifications

• 3+ years of full software development life cycle experience, including coding standards, code reviews, source control management, build processes, testing, and operations
• Bachelor’s degree in computer science or equivalent

Benefits

• Sign-on payments
• Restricted stock units (RSUs)
• Health insurance (medical, dental, vision, prescription, Basic Life & AD&D insurance, option for Supplemental life plans, EAP, Mental Health Support, Medical Advice Line)
• Flexible Spending Accounts
• Adoption and Surrogacy Reimbursement coverage
• 401(k) matching
• Paid time off
• Parental leave
• Workplace accommodation or adjustment during the application and hiring process (including support for interview or onboarding)

Additional Information for Los Angeles County Applicants

• Work safely and cooperatively with employees, supervisors, and staff
• Adhere to standards of excellence despite stressful conditions
• Communicate effectively and respectfully to ensure exceptional customer service
• Follow all federal, state, and local laws and company policies
• Criminal history may have a direct, adverse, and negative relationship with some material job duties, including the abilities to follow policies, exercise sound judgment, manage stress, work safely and respectfully with others, exhibit trustworthiness and professionalism, and safeguard business operations and the company’s reputation
• Under the Los Angeles County Fair Chance Ordinance, qualified applicants with arrest and conviction records will be considered

Location: Santa Clara, CA (onsite)

Compensation: USD 165,200 - 223,600 per year