Principal Machine Learning Engineer

Okta is building agent access authorization capabilities through Okta Secures AI, shifting from static, rule-based checks to dynamic AI security mechanisms. In this Principal Machine Learning Engineer role, you will help define a unified control plane and Identity Security Fabric for the agentic era, enabling real-time threat inspection and behavioral analysis.

Based in San Francisco, CA on a hybrid schedule, you will work on systems that enforce intent, interpret LLM signals, and integrate low-latency ML evaluation into the request path. This position is backed by a USD 238,000 - 326,000 per year salary range and is intended for experienced engineers.

What you will do

• Implement intent-based enforcement so agent runtime requests are verified against their intended purpose, supported by required key capabilities.
• Apply LLM reasoning and prompt parsing to interpret prompts, tool payloads, and intent in real time.
• Integrate low-latency inference or semantic evaluation engines directly into the API gateway request path.
• Use embeddings, vector search, or zero-shot classification to score alignment between agent intent and executed actions.
• Design confidence-scored decision engines that feed semantic verification results into policy frameworks such as Cedar.
• Establish evaluation benchmarks, prompt injection defenses, and guardrails to reduce bypasses and false positives.
• Architect scalable ML and Generative AI systems that integrate retrieval, inference, and evaluation pipelines.
• Optimize prompting, context retrieval, and RAG workflows for accuracy, safety, and efficiency in Claude-based systems.
• Build automated evaluation pipelines to measure model quality, correctness, groundedness, and safety in production.
• Implement schema validation, structured output enforcement, and guardrails to support reliable and compliant AI outputs.
• Mentor and coach engineers to support team and community growth.

What you bring

• 10+ years of software development experience with strong programming expertise in Python (familiarity with Go or TypeScript is a plus).
• Hands-on experience with applied machine learning, including feature engineering through training and fine-tuning.
• Hands-on experience with modern Generative AI platforms including AWS Bedrock, OpenAI, Anthropic, or similar.
• Deep understanding of retrieval-augmented generation (RAG), embeddings, and knowledge-base workflows.
• Hands-on experience with AI agent frameworks such as LiteLLM, LangGraph, LangChain, LlamaIndex, MCP, or related options.
• Familiarity with ML frameworks (FastAPI, PyTorch, TensorFlow, Spark ML) and workflow orchestration tools (for example, Airflow).
• Experience defining evaluation metrics, pipelines, and feedback loops for ML and GenAI systems.
• Proven ability to collaborate with product and engineering teams to drive greenfield initiatives, navigate unknowns, and iterate frequently and quickly.
• Experience building tools or infrastructure for AI/ML applications, with a strong understanding of the developer lifecycle in an AI-native world.

Technologies

• Python, Go, TypeScript, AWS Bedrock, OpenAI, Anthropic
• LiteLLM, LangGraph, LangChain, LlamaIndex, MCP
• FastAPI, PyTorch, TensorFlow, Spark ML, Airflow, Cedar, Claude-based systems

Education

Bachelor’s or Master’s degree in Computer Science or related field.

Benefits

• Equity (where applicable), bonus, and benefits including health, dental and vision insurance
• 401(k)
• Flexible spending account
• Paid leave including PTO and parental leave

Extra credit

• Experience integrating AI-driven systems with identity, authentication, or security products.
• Exposure to ethical AI, model risk, or compliance frameworks.
• Familiarity with evaluation datasets, synthetic data generation, or LLM-as-a-judge methods.

Okta values and focus

• Supporting Your Well-Being
• Driving Social Impact
• Developing Talent and Fostering Connection and Community