Sr Principal Machine Learning Engineer (Advanced Threat Protection)
Job Description
Palo Alto Networks is seeking a Sr Principal Machine Learning Engineer to drive architectural strategy and AI innovation for cloud security systems. This onsite role in Santa Clara, CA focuses on building resilient Security-as-a-Service capabilities through high-impact machine learning and distributed systems architecture.
Role Overview
As a senior principal engineering leader, you will set architectural direction, evaluate production viability of emerging AI research, and lead a multi-year roadmap for advanced threat protection. You will partner across teams to translate complex threat landscapes and product goals into executable technical designs.
Key Responsibilities
- Act as the organization’s primary technical compass by assessing frontline AI research, separating short-lived hype from production-ready technology, and directing engineering investment toward high-leverage architectural paths.
- Strengthen engineering rigor through RFC standards, architectural review boards (ARBs), and mentorship for Staff and Principal engineers to scale technical excellence across teams.
- Bridge Product Management, Security Researchers, Data Science, and Infrastructure by converting ambiguous objectives and complex threat scenarios into clear technical requirements, milestone architectures, and executable designs.
- Own macro- and micro-level architecture for highly distributed, fault-tolerant cloud systems that process real-time security data streams at petabyte scale.
- Architect production-grade agentic frameworks with dynamic multi-step planning, stateful memory persistence, secure tool execution, and deterministic guardrails.
- Lead architectural blueprints for ingestion, preprocessing, and feature engineering pipelines using Apache Spark, SQL, and BigQuery across Google Cloud platforms (Dataproc, GKE, and GCP SDKs).
- Maintain hands-on mastery of evolving model technologies and guide architectural choices across Large Language Models (LLMs), Small Language Models (SLMs), and multimodal systems to optimize cost, latency, privacy, and domain performance.
- Implement systematic frameworks to diagnose production model underperformance such as drift, context dilution, alignment collapse, and hallucination spikes; identify failure points across training data distribution, loss convergence, or prompt/context limits; and drive remediation using SFT, DPO, RL, and LoRA/PEFT.
- Troubleshoot distributed compute bottlenecks on GPU clusters, including training and inference failures, CUDA out-of-memory (OOM) conditions, NCCL communication overhead, and kernel performance stalls.
Required Qualifications
- 10+ years of progressive software engineering and systems design experience, including a documented history of technical leadership delivering high-availability, enterprise-scale platforms.
- Proven ability to author complex system designs from scratch, define data contracts, manage architectural trade-offs, and lead large-scale legacy-to-modern migrations.
- Demonstrated capability to align cross-functional teams (Product, Data, Applied Research, SecOps, Infrastructure) around unified technical visions and reconcile competing design priorities.
- Deep hands-on knowledge of modern AI architectures including Transformers, attention mechanisms, and tokenization strategies, plus operational experience deploying and serving LLMs and SLMs.
- Expert ability to analyze model degradation and execute targeted recovery pipelines involving data curation, synthetic data generation, supervised fine-tuning, preference alignment, and parameter-efficient tuning.
- Practical proficiency profiling and debugging distributed GPU workloads, including memory leak diagnosis, concurrency locks, and cluster communication bottlenecks.
- Advanced Python proficiency for AI/ML pipelines and scientific computing.
- Production-level fluency in Go or Java for low-latency, concurrent backend microservices.
- Strong data engineering foundation with Apache Spark, distributed processing, SQL, BigQuery, and GCP orchestration (Dataproc, GKE).
- BS, MS, or Ph.D. in Computer Science, Data Science, Artificial Intelligence, or a related technical discipline.
Technologies
- Python, Go, Java
- Apache Spark, SQL, BigQuery, Dataproc, GKE, GCP SDKs
- Large Language Models (LLMs), Small Language Models (SLMs), multimodal systems
- Transformers, attention mechanisms, tokenization strategies
- SFT, DPO, RL, LoRA, PEFT
- CUDA, NCCL
- Agentic frameworks, supervised fine-tuning, preference alignment, parameter-efficient tuning
Preferred Qualifications
- Deep background in threat modeling, endpoint detection and response (EDR), SIEM architectures, or cloud security posture management.
- Working knowledge of L3–L7 network protocols, packet telemetry, and deep packet inspection systems.
- Contributions to open-source foundation model tooling, published research, or active participation in the broader AI systems community.
Compensation and Location
Location: Santa Clara, CA, United States of America (onsite). Salary: USD 190,400 - 308,000 per yearly.
Compensation depends on qualifications, experience, and work location. Starting base salary (non-sales roles) or base salary + commission target (sales/commissioned roles) is expected to be $190,400.00 - $308,000.00/yr. Additional compensation may include restricted stock units and a bonus.
Immigration Sponsorship
Yes (role is eligible for immigration sponsorship).