Principal Machine Learning Engineer

Principal Machine Learning Engineer on the AI Canvas team, driving the AI layer for a next-generation cybersecurity data exploration experience.

Responsibilities

• Provide technical leadership for AI Canvas architecture and development, from early design through production deployment and continuous improvement.
• Design and build scalable, production-grade systems using LLMs, RAG, and AI agents (assistants, tools, skills, and agent harnesses).
• Architect natural-language experiences for exploring complex security data, including natural-language-to-query generation, follow-up interactions, clarification, investigation, and troubleshooting.
• Define and evolve agentic AI system architecture, covering orchestration, context management, tool invocation, memory, reasoning workflows, and multi-step task execution.
• Build robust evaluation frameworks and harnesses to measure AI quality across correctness, relevance, reliability, regression detection, and end-to-end product behavior.
• Establish evaluation methodologies using automated metrics, LLM-based evaluators, human evaluation, and representative production datasets.
• Design and implement guardrails and safety mechanisms to improve reliability, reduce hallucinations, enforce constraints, and support responsible behavior.
• Improve model and system quality through prompt engineering, retrieval strategies, model selection, fine-tuning where appropriate, and systematic experimentation.
• Build scalable RAG and knowledge-retrieval systems grounded in large, complex, and evolving security datasets.
• Partner with backend and platform engineers to deliver reliable APIs, services, and infrastructure for AI workloads at production scale.
• Establish engineering best practices for LLM and agent systems, including observability, debugging, tracing, versioning, reproducibility, testing, and monitoring.
• Evaluate emerging models, frameworks, and AI infrastructure, determining when and how to incorporate them into the product.
• Balance rapid experimentation with engineering rigor needed for mission-critical production AI systems.
• Mentor engineers, shape technical direction across teams, and raise the engineering bar for applied AI development.

Requirements

• Bachelor’s degree in Computer Science, Machine Learning, Engineering, or a related technical field, or equivalent practical experience.
• 7+ years of software engineering, machine learning engineering, or related industry experience, including significant experience building production systems.
• Strong experience designing and building machine learning or AI-powered applications at scale.
• Hands-on experience building applications using large language models and generative AI technologies.
• Experience in one or more areas such as RAG, AI agents, AI assistants, tool-calling systems, agent orchestration, or LLM-based workflows.
• Strong proficiency in Python and experience developing production-grade backend or ML services.
• Experience designing evaluation systems for AI/ML applications, including offline evaluation, regression testing, quality measurement, and production monitoring.
• Strong understanding of modern ML and AI concepts: embeddings, retrieval, ranking, prompt engineering, model inference, and experimentation.
• Experience building scalable systems on public cloud platforms such as GCP, AWS, or Azure.
• Strong software engineering fundamentals: system design, distributed systems, APIs, testing, CI/CD, and observability.
• Ability to lead complex technical initiatives across multiple teams while staying hands-on.
• Strong communication skills to translate ambiguous product problems into clear technical architectures and execution plans.

Technologies

• LLMs
• Retrieval-augmented generation (RAG)
• AI agents, AI assistants
• Natural-language-to-query generation
• Python
• GCP, AWS, Azure
• Embeddings
• Prompt engineering
• CI/CD
• Docker, Kubernetes
• Vector databases
• Natural-language-to-SQL
• LLM-based evaluators

Preferred Experience

• Master’s or PhD in Computer Science, Machine Learning, Artificial Intelligence, or a related technical field.
• Deep experience building and operating LLM-powered products or agentic AI platforms in production.
• Experience with AI development frameworks and tooling for model orchestration, agent systems, evaluation, tracing, or observability.
• Experience building natural-language-to-SQL or natural-language-to-query systems, semantic layers, or AI-powered data exploration products.
• Experience designing LLM evaluation harnesses, synthetic test generation, LLM-as-a-judge approaches, or human-in-the-loop workflows.
• Experience with vector databases, search and retrieval infrastructure, embeddings, and large-scale knowledge systems.
• Experience with containerization and orchestration technologies such as Docker and Kubernetes.
• Experience designing highly available, low-latency AI inference and backend services.
• Familiarity with AI security: prompt injection defenses, data privacy, access control, and guardrails for enterprise AI.
• Experience in cybersecurity, security analytics, observability, or large-scale data platforms.
• Contributions to open-source AI, ML, agent, or infrastructure projects.

Location

• Santa Clara, CA (onsite)

Compensation

• $163,200.00 - $264,000.00/yr
• Compensation depends on qualifications, experience, and work location.
• For offers at the posted level, starting base salary for non-sales roles or base salary plus commission target for sales/commissioned roles is expected to fall within the listed range.
• May include restricted stock units and a bonus.

• Immigration sponsorship eligibility: Yes