EngineerJobs.io
← Back to all jobs

Job Description

Palo Alto Networks is seeking a Sr Principal Machine Learning Engineer to drive architectural strategy and AI innovation for cloud security systems. This onsite role in Santa Clara, CA focuses on building resilient Security-as-a-Service capabilities through high-impact machine learning and distributed systems architecture.

Role Overview

As a senior principal engineering leader, you will set architectural direction, evaluate production viability of emerging AI research, and lead a multi-year roadmap for advanced threat protection. You will partner across teams to translate complex threat landscapes and product goals into executable technical designs.

Key Responsibilities

  • Act as the organization’s primary technical compass by assessing frontline AI research, separating short-lived hype from production-ready technology, and directing engineering investment toward high-leverage architectural paths.
  • Strengthen engineering rigor through RFC standards, architectural review boards (ARBs), and mentorship for Staff and Principal engineers to scale technical excellence across teams.
  • Bridge Product Management, Security Researchers, Data Science, and Infrastructure by converting ambiguous objectives and complex threat scenarios into clear technical requirements, milestone architectures, and executable designs.
  • Own macro- and micro-level architecture for highly distributed, fault-tolerant cloud systems that process real-time security data streams at petabyte scale.
  • Architect production-grade agentic frameworks with dynamic multi-step planning, stateful memory persistence, secure tool execution, and deterministic guardrails.
  • Lead architectural blueprints for ingestion, preprocessing, and feature engineering pipelines using Apache Spark, SQL, and BigQuery across Google Cloud platforms (Dataproc, GKE, and GCP SDKs).
  • Maintain hands-on mastery of evolving model technologies and guide architectural choices across Large Language Models (LLMs), Small Language Models (SLMs), and multimodal systems to optimize cost, latency, privacy, and domain performance.
  • Implement systematic frameworks to diagnose production model underperformance such as drift, context dilution, alignment collapse, and hallucination spikes; identify failure points across training data distribution, loss convergence, or prompt/context limits; and drive remediation using SFT, DPO, RL, and LoRA/PEFT.
  • Troubleshoot distributed compute bottlenecks on GPU clusters, including training and inference failures, CUDA out-of-memory (OOM) conditions, NCCL communication overhead, and kernel performance stalls.

Required Qualifications

  • 10+ years of progressive software engineering and systems design experience, including a documented history of technical leadership delivering high-availability, enterprise-scale platforms.
  • Proven ability to author complex system designs from scratch, define data contracts, manage architectural trade-offs, and lead large-scale legacy-to-modern migrations.
  • Demonstrated capability to align cross-functional teams (Product, Data, Applied Research, SecOps, Infrastructure) around unified technical visions and reconcile competing design priorities.
  • Deep hands-on knowledge of modern AI architectures including Transformers, attention mechanisms, and tokenization strategies, plus operational experience deploying and serving LLMs and SLMs.
  • Expert ability to analyze model degradation and execute targeted recovery pipelines involving data curation, synthetic data generation, supervised fine-tuning, preference alignment, and parameter-efficient tuning.
  • Practical proficiency profiling and debugging distributed GPU workloads, including memory leak diagnosis, concurrency locks, and cluster communication bottlenecks.
  • Advanced Python proficiency for AI/ML pipelines and scientific computing.
  • Production-level fluency in Go or Java for low-latency, concurrent backend microservices.
  • Strong data engineering foundation with Apache Spark, distributed processing, SQL, BigQuery, and GCP orchestration (Dataproc, GKE).
  • BS, MS, or Ph.D. in Computer Science, Data Science, Artificial Intelligence, or a related technical discipline.

Technologies

  • Python, Go, Java
  • Apache Spark, SQL, BigQuery, Dataproc, GKE, GCP SDKs
  • Large Language Models (LLMs), Small Language Models (SLMs), multimodal systems
  • Transformers, attention mechanisms, tokenization strategies
  • SFT, DPO, RL, LoRA, PEFT
  • CUDA, NCCL
  • Agentic frameworks, supervised fine-tuning, preference alignment, parameter-efficient tuning

Preferred Qualifications

  • Deep background in threat modeling, endpoint detection and response (EDR), SIEM architectures, or cloud security posture management.
  • Working knowledge of L3–L7 network protocols, packet telemetry, and deep packet inspection systems.
  • Contributions to open-source foundation model tooling, published research, or active participation in the broader AI systems community.

Compensation and Location

Location: Santa Clara, CA, United States of America (onsite). Salary: USD 190,400 - 308,000 per yearly.

Compensation depends on qualifications, experience, and work location. Starting base salary (non-sales roles) or base salary + commission target (sales/commissioned roles) is expected to be $190,400.00 - $308,000.00/yr. Additional compensation may include restricted stock units and a bonus.

Immigration Sponsorship

Yes (role is eligible for immigration sponsorship).

Similar Jobs