Lead Principal Machine Learning Engineer

The Lead Principal Machine Learning Engineer at Oracle leads the design, development, and operation of next generation AI systems on Oracle Cloud Infrastructure. Based onsite in San Francisco, CA, this senior technical role requires a PhD and at least 12 years of relevant experience, directing architecture and engineering for enterprise AI platforms, autonomous workflows, scalable inference, and business‑critical AI applications deployed at scale.

Responsibilities

• Assume senior technical ownership of OCI AI platform capabilities, including agent execution, inference services, model serving, AI workflow orchestration, evaluation, and observability.
• Design and deliver scalable agentic AI systems capable of reasoning, planning, tool use, workflow execution, multi-step task orchestration, and safe human-in-the-loop escalation.
• Develop production grade services for tool invocation, agent memory, context management, Model Context Protocol integration, vector retrieval, multi-agent coordination, policy enforcement, and evaluation.
• Direct architecture across distributed services with a focus on low latency, high throughput, GPU efficiency, reliability, cost control, operability, and secure multi-tenant operation.
• Define service boundaries, APIs, data models, state management, consistency tradeoffs, failure modes, SLIs/SLOs, rollout strategies, and readiness criteria for AI platform services.
• Shape technical strategy across infrastructure, platform, security, data, and application teams, translating broad goals into executable multi-quarter plans with clear milestones.
• Integrate AI agents securely and reliably with enterprise APIs, cloud services, databases, identity systems, secrets management, and external systems.
• Establish AgentOps and LLMOps practices for tracing, monitoring, evaluation suites, regression testing, experimentation, safety guardrails, prompt/tool versioning, and production reliability.
• Evaluate and operationalize emerging technologies in generative AI, agentic workflows, inference optimization, long-context systems, reasoning models, AI developer tooling, and agent-first development.
• Promote engineering excellence through code and design reviews, test strategy, deployment automation, incident analysis, documentation, and AI-assisted development using tools such as Codex, Claude Code, Cursor, Copilot, or equivalents.
• Mentor staff and senior engineers, elevate architectural standards, and influence OCI engineering practices without direct management authority.
• Own critical production outcomes including reliability, performance, security posture, cost efficiency, and maintainability of delivered systems.

Requirements

• PhD in Computer Science, AI/ML, Engineering, or a related field, or equivalent practical experience.
• 12+ years of professional software engineering experience with substantial ownership of production systems, or equivalent impact at senior staff or principal levels.
• Proven track record as a Staff, Senior Staff, Principal, or equivalent technical leader influencing architecture and execution across multiple teams.
• Deep experience designing, building, and operating high-scale distributed systems, cloud services, infrastructure platforms, or AI/ML platform services.
• Hands-on experience with production AI systems, agentic AI applications, autonomous workflows, tool-using agents, multi-step orchestration, or multi-agent systems.
• Practical experience with orchestration frameworks such as LangGraph, LangChain, CrewAI, AutoGen, LlamaIndex, or similar ecosystems.
• Deep understanding of LLM application patterns, including prompt design, structured outputs, function/tool calling, context management, RAG, memory, tool safety, and evaluation.
• Strong Python programming skills with the ability to contribute production code, reviews, tests, and debugging in complex distributed environments.
• Extensive expertise with Kubernetes, Docker, cloud-native infrastructure, service-to-service communication, scalability, fault tolerance, observability, and performance analysis.
• Experience defining SLIs/SLOs, production readiness criteria, incident response practices, monitoring, tracing, experiments, and reliability programs for AI or distributed systems.
• Solid understanding of AI safety, governance, security, and operational risks for autonomous or semi-autonomous systems, including data handling, access control, auditability, and human accountability.
• Excellent written and verbal communication, with demonstrated ability to lead technical direction, resolve ambiguity, and influence senior stakeholders.

Technologies

• Python
• Kubernetes
• Docker
• LangGraph
• LangChain
• CrewAI
• AutoGen
• LlamaIndex
• Model Context Protocol (MCP)
• Oracle Cloud Infrastructure (OCI)
• Codex
• Claude Code
• Cursor
• Copilot

Preferred Qualifications

• Experience optimizing large-scale GPU inference or training workloads for latency, throughput, utilization, availability, and cost.
• Experience building or operating model serving components, inference gateways, agent runtimes, workflow engines, developer platforms, or internal AI productivity platforms.
• Experience integrating AI systems with enterprise APIs, databases, cloud services, vector databases, embeddings, retrieval systems, identity systems, and policy enforcement layers.
• Experience with LLM fine-tuning, long-context systems, reasoning models, model routing, caching, batching, quantization, or emerging generative AI research.
• Experience building evaluation frameworks for agentic systems, including offline evals, online experiments, golden tasks, adversarial testing, regression gates, and observability dashboards.
• Experience using AI-assisted software development tools such as Codex, Claude Code, Cursor, Copilot, or similar systems in large-scale engineering environments.
• Track record of defining architectural standards, platform capabilities, or engineering practices adopted across multiple teams or organizations.
• Experience in enterprise, cloud infrastructure, regulated, security-sensitive, or mission-critical environments.

Location

San Francisco, CA (onsite)