Genai Moderation Engineer Jobs

1 job found

Today

AI Engineer

News Corp Austin, TX

Austin, TX

Today

Full time Ai Security Ai Security Evaluation Artificial Intelligence

Apply Now

Job Description

News Corp is looking for an AI Engineer to join Realtor.com’s AI Integrations Team in Austin, TX on an onsite basis. In this role, you will build and tune LLM safety guardrails for RealAssist, owning the AI safety layer end-to-end from design through deployment.

Because RealAssist advises consumers on housing, the system must meet Fair Housing Act compliance obligations. This position treats AI safety as a launch gate, supported by evaluation pipelines, runtime defenses, and ongoing red-teaming to keep guardrails effective as the product evolves.

What you’ll do

Build, tune, and own LLM-as-a-judge classifiers for Fair-Housing compliance and content moderation, targeting high recall on disallowed content without overblocking legitimate users.
Design and run guardrail evaluation pipelines, including curating labeled datasets, maintaining train/test/validate splits, and executing offline prod-replay overblocking evals.
Produce confusion matrices and precision/recall/F1 per category to support defensible safety decisions.
Integrate and operate cloud guardrail services as a runtime prompt-injection and jailbreak screening layer, including fail-open behavior and alerting.
Red-team the assistant and wire safety regression checks into CI so guardrails do not degrade silently as the product grows.
Manage guardrail infrastructure as code using Terraform (templates, IAM, project shape), and participate in technical design reviews and architecture discussions.
Collaborate with ML, backend, product, and legal/compliance stakeholders to define risk tiering and route new AI capabilities through safety review before launch.
Independently design, implement, and tune guardrail and evaluation systems, balancing safety, latency, and user experience, with a focus on clean, maintainable, well-tested code.
Keep evaluation datasets calibrated to real traffic and emerging attack patterns, stay current with AI safety research and tooling, and make data-driven decisions grounded in evaluation metrics.
Deliver high-quality work consistently on schedule.

Required qualifications

4+ years of professional software/ML engineering experience with hands-on LLM application work.
AI Safety & Evaluation Expertise (Required).
Strong Python proficiency.
Real classification-metrics literacy, including precision/recall/F1 (macro vs weighted), confusion matrix debugging, dataset curation, and calibration to production distributions.
Experience building or tuning LLM prompts/classifiers and evaluating them with an eval framework (examples include DeepEval, RAGAS, Arize Phoenix, LangSmith, OpenAI Evals, or a home grown harness).
Experience turning small seed sets into robust labeled eval datasets, including dataset-synthesis tooling and prompt-optimization loops.
Familiarity with prompt-injection/jailbreak defense concepts, including OWASP LLM Top 10, input/output filtering, least-privilege tool access, and adversarial testing.
Hands-on experience with a cloud guardrail service, such as Google Cloud Model Armor, AWS Bedrock Guardrails, or Azure AI Content Safety.
Terraform / IaC for cloud infrastructure.
Experience with Google Cloud Vertex AI / Gemini.
Understanding of monitoring and observability tools (for example, New Relic or similar).
Exposure to regulated or compliance-sensitive domains including fair housing, fair lending, healthcare, finance, or trust & safety.
Red-teaming or AI-security background.
Bachelor’s degree or equivalent experience.

Technologies

Python
DeepEval, RAGAS, Arize Phoenix, LangSmith, OpenAI Evals
OWASP LLM Top 10
Google Cloud Model Armor, AWS Bedrock Guardrails, Azure AI Content Safety
Terraform
Google Cloud Vertex AI / Gemini
New Relic

Benefits

Inclusive and competitive medical, Rx, dental, and vision coverage
Family forming benefits
13 Paid Holidays
Flexible Time Off
8 hours of paid Volunteer Time off
Immediate eligibility into the Company 401(k) plan with 3.5% company match
Tuition Reimbursement program for degreed and non-degreed programs
1:1 personalized Financial Planning Sessions
Student Debt Retirement Savings Match program
Free snacks and refreshments in each office location

Role context: The AI Integrations Team is a cross-functional squad building how Realtor.com integrates AI across its products. RealAssist is an LLM-powered real-estate assistant embedded across web and mobile, where the safety layer serves as a critical launch gate for compliance.

View full job page →

Get Genai Moderation Engineer Jobs alerts

New Genai Moderation Engineer Jobs, daily.