At Apple, the Cloud AI Platform team is building the systems that enable large-scale machine learning across teams and use cases. In this Senior / Staff Machine Learning Engineer role, you will help shape technical direction while developing training recipes, debugging stability and performance at scale, and creating scalable training and post-training infrastructure.
This position is based in Seattle, WA (onsite), with a base pay range of USD 175,000 - 308,500 per year. The role expects 5+ years of relevant experience and a BS, MS, or PhD degree in Computer Science, Computer Engineering, or equivalent practical experience.
What you’ll do
- Partner with researchers and ML engineers to develop advanced models and translate their needs into training features and performance improvements usable across the platform.
- Create model distillation and compression approaches, including teacher-student distillation and quantization-aware training, to bring large-model quality to smaller models.
- Train models that use long-context extension, mixture-of-experts architectures, and multimodal inputs, while addressing the stability and efficiency challenges each approach introduces.
- Investigate large-scale training behavior such as loss spikes, divergence, and throughput drops, tracing issues back to root causes across data, numerics, code, or hardware.
- Build and own scalable, efficient training frameworks and services on GPUs and TPUs, leveraging PyTorch or JAX along with higher-level frameworks.
- Develop reinforcement learning infrastructure for post-training, including rollout and inference services, reward pipelines, and secure sandboxes that allow models to run code and use tools.
- Integrate the training platform with Apple’s data and governance stack so each training run meets privacy and security standards.
- Set technical direction across teams, mentor engineers, and raise the engineering bar.
Experience and qualifications
- Experience with low-precision training and the numerical issues it introduces.
- Experience training large models using data, tensor, pipeline, or expert parallelism across hundreds to thousands of GPUs or TPUs.
- Strong understanding of GPU and TPU architecture, including memory hierarchy, matrix units, and high-bandwidth memory, and how these affect training performance.
- Experience with collective communication libraries such as NCCL and high-speed interconnects such as NVLink, InfiniBand, and TPU ICI.
- Experience profiling and optimizing training workloads using tools such as Nsight Systems, PyTorch Profiler, or XProf.
- Experience building reinforcement learning or post-training systems for large models.
- Open source contributions to training frameworks and/or publications at top ML venues such as NeurIPS, ICML, or ICLR.
- 8+ years of experience in large-scale model training, including reinforcement learning, from research to production.
- 5+ years of experience building and operating large-scale model training systems in production.
- Hands-on experience with fine-tuning and distillation methods such as supervised fine-tuning, parameter-efficient fine-tuning with LoRA, and teacher-student distillation.
- Hands-on experience with at least one higher-level training framework such as Megatron-LM, DeepSpeed, TorchTitan, NeMo, AXLearn, or MaxText.
- Deep expertise in PyTorch or JAX, including distributed training features.
- Strong programming skills in Python or C++, plus strong written and verbal communication skills and a record of leading technical work across teams.
- BS, MS, or PhD in Computer Science, Computer Engineering, or equivalent practical experience.
Technologies you may work with
- PyTorch, JAX
- Parameter-efficient fine-tuning (LoRA), teacher-student distillation, quantization-aware training
- Supervised fine-tuning
- Megatron-LM, DeepSpeed, TorchTitan, NeMo, AXLearn, MaxText
- NCCL, NVLink, InfiniBand, TPU ICI
- Nsight Systems, PyTorch Profiler, XProf
- Python, C++
Benefits
- Comprehensive medical and dental coverage
- Retirement benefits
- A range of discounted products and free services
- Reimbursement for certain educational expenses, including tuition for formal education related to advancing your career
- Participation in Apple’s discretionary employee stock programs, including discretionary restricted stock unit awards and the Employee Stock Purchase Plan
- Discretionary bonuses or commission payments, as applicable
- Relocation, as applicable
Note: Apple benefit, compensation, and employee stock programs are subject to eligibility requirements and other terms of the applicable plan or program. Base pay range for this role is between $175,000 and $308,500, depending on skills, qualifications, experience, and location.
