Tech Lead Machine Learning Engineer - LLM Evaluation & Agent Systems

Build responsible AI systems with a team focused on safety, integrity, and compliance. In this onsite Tech Lead role, you will drive architecture and execution for an AI Risk & Compliance Intelligence Platform and an Intelligent Data Agent Ecosystem. The work centers on LLM evaluation, synthetic data generation, and modular agent optimization for Data/Engineering/SRE workflows.

This position is based in Seattle, WA (onsite) and offers USD 198,360 - 416,100 per year. You will partner across cross-functional teams to translate operational requirements into scalable machine learning solutions.

Responsibilities

• Design, build, and scale robust LLM evaluation architecture, including automated offline datasets, metric suites, and benchmarking pipelines to measure prompt and model performance.
• Innovate on LLM-driven synthetic data generation to address data scarcity bottlenecks and systematically evaluate long-tail and edge-case risk scenarios.
• Move beyond criteria-based prompt engineering by extracting deep risk patterns from complex case data to create native, ML-driven decision models and reasoning abstractions.
• Architect, evaluate, and continuously optimize modular agent skills and tools for Data Science, Analytics, Data Engineering, and SRE workflows to improve tool-use accuracy, execution efficiency, and task success rates.
• Implement advanced evaluation methodologies, including LLM-as-a-judge and multi-agent benchmarking, across both risk intelligence and agentic ecosystems.
• Provide technical leadership by defining core ML system design patterns, benchmarking standards, and engineering best practices, while translating complex operational requirements into scalable ML solutions with cross-functional partners.

Requirements

• 5+ years of software/ML engineering experience with a strong record of designing, building, and deploying ML/LLM systems in production.
• Deep hands-on expertise with LLM architectures, fine-tuning, RAG, prompt tuning, and agentic frameworks.
• Demonstrated experience building robust LLM evaluation frameworks (LLM-as-a-judge, automated benchmark pipelines) and addressing data scarcity/imbalance.
• Strong coding skills with production experience in scalable data processing pipelines.
• Prior domain experience in Risk & Compliance Intelligence and Automated Decision Systems.
• Hands-on experience with synthetic dataset generation or bootstrapping models for cold-start scenarios.
• Experience building AI tools or agents for technical personas including Data Analysts, Data Scientists, Data Engineers, and Data SRE.
• Ability to serve as a founding technical lead on ambiguous, high-impact initiative pods.

Benefits

• Day one access to medical, dental, and vision insurance
• 401(k) savings plan with company match
• Paid parental leave
• Short-term and long-term disability coverage
• Life insurance
• Wellbeing benefits, among others
• 10 paid holidays per year
• 10 paid sick days per year
• 17 days of Paid Personal Time (prorated upon hire with increasing accruals by tenure)

About the Team

The TikTok U.S. Data Security (USDS) team is responsible for the security, integrity, and compliance of TikTok’s data within the United States. The Data Platform Team builds scalable, reliable, cost-efficient infrastructure that supports data integrity and empowers business units through self-service tools and scientific insights.

About USDS

TikTok USDS Joint Venture LLC is dedicated to the safety and security of millions of Americans who create, discover, and connect with what they love on the apps operated by TikTok. USDS maintains onsite presence across teams to support speed, alignment, and agility, and the organization is shifting from a hybrid model to a fully in-person schedule up to 5 days a week in the areas outlined in the role.

USDS Reasonable Accommodation

USDS is committed to providing reasonable accommodations in recruitment processes for candidates with disabilities, pregnancy, sincerely held religious beliefs, or other reasons protected by applicable laws.