Distributed Training

Distributed Training Engineer Jobs

4 jobs found

9d ago

Machine Learning Engineer

Unity Technologies SF Mountain View, CA

Mountain View, CA

9d ago

$104k - $152k/yr Full time Apache Airflow Artificial Intelligence Automation

Apply Now

Job Description

Machine Learning Engineer in Mountain View, CA (onsite) on Unity's Offline Infrastructure team, building data pipelines for training data generation and distributed model training to scale ML workflows.

Salary

USD 104,100 - 152,000 per year

Responsibilities

Design and maintain data pipelines that generate training datasets for ML models and experiments
Contribute to infrastructure supporting distributed training workflows using PyTorch and Ray
Collaborate with workflow orchestration tools like Airflow or Flyte to support multi-stage ML pipelines
Enhance reproducibility and reliability through dataset validation, monitoring, and testing
Partner with ML engineers to back experimentation and model iteration
Optimize performance and efficiency across data processing and training systems
Contribute to the evolution of the offline ML platform architecture as it scales

Requirements

Bachelor's degree in Computer Science, Machine Learning, Systems, or a related field
Strong foundation in ML systems, distributed systems, or large-scale data processing demonstrated through research or projects
Experience with Python and working with data-intensive workloads
Familiarity with ML frameworks (PyTorch, TensorFlow) and/or distributed systems (Ray, Spark)
Experience (academic or applied) with data pipelines, model training workflows, or large datasets
Strong problem-solving skills and ability to translate research ideas into practical systems
Interest in building scalable, reliable infrastructure for machine learning

Technologies

Python
PyTorch
TensorFlow
Ray
Spark
Airflow
Flyte

Benefits

Comprehensive health, life, and disability insurance
Commute subsidy
Employee stock ownership
Competitive retirement/pension plans
Generous vacation and personal days
Support for new parents through leave and family-care programs
Office snacks
Mental Health and Wellbeing programs and support
Employee Resource Groups
Global Employee Assistance Program
Training and development programs
Volunteering and donation matching program

Additional Information

Relocation assistance is not available for this position
Work visa/immigration sponsorship is not available for this position

View full job page →

10d ago

Senior Machine Learning Engineer

Placement Force Remote 🌍 Remote

Remote

10d ago

$160k - $260k/yr Full time Senior Artificial Intelligence Data Pipeline

Apply Now

Job Description

Placement Force is seeking a Senior Machine Learning Engineer to own the complete lifecycle of a production large language model. The role blends hands-on model training, scalable deployment, and thorough evaluation to ensure the shipped system meets real-world objectives. This is a remote position that demands a proven ability to deliver robust ML systems in production environments.

Location: Remote

Compensation: USD 160,000 - 260,000 per year

Minimum experience: 4 years

Responsibilities

Architect and run pretraining, continued pretraining, and fine-tuning experiments, including SFT, DPO/RLHF-style alignment, LoRA/QLoRA, and full-parameter methods, all tied to clear, measurable objectives
Own data pipeline decisions that meaningfully impact model quality, covering data curation, de-duplication, mixture weighting, and contamination checks against evaluation sets
Conduct and interpret distributed training across multi-GPU and multi-node setups using frameworks such as FSDP, DeepSpeed, or Megatron-style parallelism; diagnose failures that appear only at scale
Make and defend practical tradeoffs between model size, training cost, and downstream performance
Take trained models to production with quantization, batching strategies, KV-cache management, and serving framework choices (for example vLLM, TensorRT-LLM, or TGI) while meeting explicit latency, throughput, and cost targets
Design for LLM serving failure modes, including tail latency under load, graceful degradation, prompt injection surfaces, and safe fallback behavior
Establish and maintain production-grade operational practices such as monitoring, alerting, and rollback procedures, treating the model as a critical service rather than a one-off export
Develop evaluation harnesses that go beyond published benchmarks, incorporating task-specific evaluations aligned with real product use cases
Design human evaluation protocols to complement automated metrics and determine when each approach is appropriate
Own regression detection to catch quality drops caused by new checkpoints, prompt template changes, or serving optimizations before users notice
Contribute to safety and robustness assessments, including measuring hallucination rates, adversarial testing, and behavior under distribution shift as an integral part of the release process

Requirements

At least 4 years of applied ML engineering experience, with a minimum of 2 years working directly on large language models in production
Hands-on production deployment experience, including shipping a model that served live traffic and being able to discuss latency, cost, and quality tradeoffs
Strong software engineering fundamentals
Fluency with the modern LLM tooling landscape, including training frameworks, serving frameworks, and evaluation tooling
Comfort with ambiguity and the ability to define what constitutes “good” model behavior in areas lacking established benchmarks

Technologies

FSDP
DeepSpeed
Megatron-style parallelism
vLLM
TensorRT-LLM
TGI

View full job page →

1mo ago

Machine Learning Engineer, Commerce Ads Ranking

TikTok San Jose, CA

San Jose, CA

1mo ago Closed

$156k - $317k/yr Full time Ad Tech Ai Ml Distributed Training

This position is no longer accepting applications (closed Aug 10, 2026).

Job Description

This on-site role focuses on optimizing models across the TikTok Commerce Ads Ranking pipeline, from recall through final ranking, and on algorithm and system co-design plus large-scale distributed training within privacy-preserving environments. The position is based in San Jose, California.

Responsibilities

Optimizing models in the ads delivery system, including feature engineering, model architecture, auto crossing, ads cold start handling, modeling delayed feedback, multi-task learning, and sequence modeling.
Algorithm and system co-design, covering retrieval algorithms, sample mining, and long sequence processing.
Exploring large-scale distributed training frameworks with focus on GPU tuning, feature processing, and synchronous training to enhance ads delivery efficiency in privacy-preserving environments.
LRM, the next generation recommender system leveraging an LLM learning paradigm: entity understanding, end-to-end Generative Recommendation, sequence-only Recommendation, and a Mixture of Export approach.

Requirements

Strong coding skills in C++ and Python, with a solid foundation in data structures and algorithms.
Solid foundation in machine learning, familiarity with common algorithmic models such as FM, LR, GBDT, DNN, Wide & Deep, etc.
Excellent learning ability and good teamwork spirit, with strong analytical and problem-solving skills. Publications in conferences such as KDD, NeurIPS, SIGIR, etc., or experience in data mining or machine learning competitions are a good fit.
Bachelor’s degree or higher in computer science or related majors.

Technologies

C++
Python
LLM

Benefits

Medical, dental, and vision insurance from day one
401(k) savings plan with company match
Paid parental leave
Short-term and long-term disability coverage
Life insurance
Wellbeing benefits
10 paid holidays per year
10 paid sick days per year
17 days of Paid Personal Time, prorated at hire with accruals increasing with tenure
Eligible for discretionary bonuses or incentives
Restricted stock units

Los Angeles County (Unincorporated) Candidates

Interacting and occasionally having unsupervised contact with internal and external clients or colleagues
Appropriately handling and managing confidential information, including proprietary and trade secret information and access to information technology systems
Exercising sound judgment

View full job page →

1mo ago

Software Engineer- AI/ML, AWS Neuron Distributed Training - Performance Optimization

Annapurna Labs (U.S.) Inc. Cupertino, CA

Cupertino, CA

1mo ago Closed

$144k - $194k/yr Full time AWS Aws Neuron Cloud

This position is no longer accepting applications (closed Jul 21, 2026).

Job Description

Software Engineer on the AWS Neuron Distributed Training team focused on performance optimization for distributed training on Trainium, collaborating with PyTorch, JAX, and Neuron compiler/runtime to improve throughput and efficiency for large-scale models.

Location and compensation details: Cupertino, CA onsite. Salary range of USD 143,700 to 194,400 per year. Minimum experience: 3+ years. Education: Bachelor's degree in computer science or equivalent.

Responsibilities

Lead efforts to optimize distributed training performance on Trainium, with a primary focus on maximizing training throughput, model FLOPs utilization, and overall efficiency across the Neuron software stack.
Collaborate across PyTorch, JAX, and the Neuron compiler and runtime to enable and tune large-scale training workloads on the latest Trainium instances.

Requirements

3+ years of non-internship professional software development experience
2+ years of non-internship design or architecture experience (design patterns, reliability, and scaling) of new and existing systems
Experience programming with at least one software programming language
3+ years of full software development life cycle experience, including coding standards, code reviews, source control management, build processes, testing, and operations
Bachelor's degree in computer science or equivalent

Technologies

PyTorch
JAX
AWS Neuron
Neuron compiler
Neuron runtime
Trainium
Trn3/Trn2/Trn1
Inf2/Inf1 servers

Benefits

Health insurance
401(k) matching
Paid time off
Parental leave
Sign-on payments
Restricted stock units (RSUs)
Basic Life & AD&D insurance
Supplemental life plans
Employee Assistance Program (EAP)
Mental health support
Medical advice line
Flexible Spending Accounts
Adoption and surrogacy reimbursement coverage

View full job page →

Distributed Training Engineer Jobs

Machine Learning Engineer

Senior Machine Learning Engineer

Machine Learning Engineer, Commerce Ads Ranking

Software Engineer- AI/ML, AWS Neuron Distributed Training - Performance Optimization

Get Distributed Training Engineer Jobs alerts