EngineerJobs.io
← Back to all jobs

Job Description

Palo Alto Networks is seeking a Principal Machine Learning Engineer to provide technical leadership in architecting and delivering next-generation, AI-driven cloud security solutions. The role focuses on security telemetry and Security-as-a-Service (SECaaS), guiding systems from prototype to enterprise-grade production.

Key Responsibilities

  • Lead the design and implementation of ML and GenAI-powered agents that support dynamic planning, multi-step execution, self-correction, and seamless collaboration across multi-agent and human-in-the-loop workflows.
  • Build, evaluate, and fine-tune both LLMs and SLMs, including multimodal architectures.
  • Diagnose model regressions or sub-par performance, identify root causes such as data distribution shift, alignment failures, and loss anomalies, and apply targeted fine-tuning approaches including SFT, DPO/RLHF, and PEFT/LoRA to drive continuous quality improvement.
  • Profile, debug, and optimize model training and inference on GPU clusters, addressing memory allocation bottlenecks (OOMs), distributed communication overhead, and compute utilization.
  • Architect scalable data preprocessing and feature extraction pipelines to support real-time endpoint telemetry feeding into agentic and analytical workflows.
  • Drive end-to-end engineering from prototype through enterprise-grade production, establishing best practices across prompt architecture, agent memory persistence, context window optimization, tool-use execution, and safety guardrails.
  • Continuously evaluate state-of-the-art agent frameworks, model serving engines, and accelerator architectures to inform long-term technical roadmaps.

Required Qualifications

  • 7+ years of professional software engineering experience with a demonstrated ability to deploy production-grade ML, GenAI, or agentic architectures at enterprise scale.
  • Advanced proficiency in Python for AI/ML development, along with production-level Go or Java experience for distributed microservices.
  • Strong hands-on capability building and scaling data pipelines.
  • Proficiency with Apache Spark, core data tools, SQL, BigQuery schema design, and Google Cloud infrastructure including Dataproc, GKE, and GCP SDKs.
  • Practical operational experience with LLMs and SLMs.
  • Deep expertise diagnosing model performance issues and executing remedial fine-tuning strategies, including dataset curation, parameter-efficient fine-tuning, loss monitoring, alignment, and evaluation benchmarks.
  • Practical experience with GPU profiling and debugging (including CUDA memory optimization, multi-GPU synchronization, batch throughput tuning, and handling GPU compute crashes or stalls).
  • Solid understanding of Retrieval-Augmented Generation (RAG), vector storage, prompt optimization, model evaluation metrics, and applied statistical modeling.
  • Bachelor’s or Master’s degree in Computer Science, Data Science, Software Engineering, or a related technical discipline.

Technologies

  • Python, Go, Java
  • Apache Spark, SQL, BigQuery
  • Google Cloud, Dataproc, GKE, GCP SDKs
  • LLMs, SLMs
  • Retrieval-Augmented Generation (RAG), vector storage
  • SFT, DPO/RLHF, PEFT/LoRA
  • CUDA, GPU clusters, multi-GPU synchronization

Preferred Qualifications (Bonus)

  • Background in cybersecurity, threat detection, or cloud security principles.
  • Working knowledge of L3–L7 networking protocols, packet inspection, and network security topologies.
  • Experience handling high-volume cybersecurity telemetry, endpoint detection data, or real-time threat intelligence feeds.

Compensation

The compensation offered for this position depends on qualifications, experience, and work location. For candidates receiving an offer at the posted level, the expected annual range is $163,200.00 - $264,000.00/yr. The offered compensation may also include restricted stock units and a bonus.

Location and Work Arrangement

Santa Clara, CA (onsite)

Company Commitments

  • Reasonable accommodations are available for qualified individuals with a disability.
  • Palo Alto Networks is an equal opportunity employer.

Similar Jobs