Llm Evaluations Engineer Jobs

1 job found

Today

Manager, Global Private Equity Data and AI Engineer

The Carlyle Group Washington, DC

Washington, DC

Today

Full time Backend Developer Manager Artificial Intelligence

Apply Now

Job Description

Lead hands-on delivery of production AI and governed data services for Global Private Equity, supporting investment professionals with natural-language access to trusted portfolio data.

Responsibilities

Design, build and deploy LLM-powered tools, agents and integrations for investment, valuation and fund finance teams to work with governed portfolio data in natural language.
Develop solutions that include Model Context Protocol (MCP) servers and integrations with the firm’s enterprise AI platforms and assistants, owning each initiative end-to-end from discovery through production and adoption.
Define evaluation cases, regression suites and acceptance criteria for AI features; track accuracy, cost and latency; gate releases on results.
Establish architectural boundaries between deterministic calculation and validation versus model judgment, ensuring answers trace back to source data.
Implement controls for governed access and safety, including OAuth/OIDC authentication, entitlement-based row-level security, SQL and tool guardrails, handling of untrusted content and prompt injection, plus audit logging.
Partner with Information Security, Compliance and Data Governance to support secure and auditable AI and data services.
Lead and mentor a team of contract developers, data engineers and QA: set technical direction, review designs and code, make build-versus-buy recommendations, and remain hands-on in the codebase.
Maintain governed data models and metadata, including metric definitions, glossaries and lineage, to support a single source of truth across reporting, applications and AI consumers.
Promote code review and automated testing across unit tests, dbt tests, data reconciliation and AI evaluations.
Operate CI/CD using GitHub Actions, apply infrastructure as code with Terraform on AWS, and manage disciplined promotion across development, QA, UAT and production.
Own production stability for AI services and the underlying data pipelines, including incident response, root cause analysis and remediation, with monitoring, alerting and runbooks.
Ensure accurate, auditable data for valuations and performance metrics (including IRR, MOIC, DPI and NAV) and investor reporting; support audit and control reviews.
Support transition of remaining legacy data and reporting workloads to the cloud platform, coordinating with vendor delivery teams.
Adopt AI coding assistants and agent-based workflows with human review and safety guardrails, sharing team knowledge (rules, skills and documentation) to improve delivery speed and quality.
Work with product owners and business users to convert ambiguous workflows into a prioritized backlog; deliver using Agile in Jira; communicate progress, risks, dependencies and technical decisions to senior management and stakeholders.

Requirements

Bachelor’s degree (required).
Concentration in computer science, engineering, finance, economics or another quantitative discipline (preferred).
Cloud, data platform or AI certification (examples: AWS, Microsoft Azure, SnowPro, dbt) (preferred).
7+ years overall relevant experience in software engineering, data engineering or technical analytics (required).
3+ years building and operating production software or data solutions used by business stakeholders (required).
2+ years leading engineering delivery, including owning a backlog, setting technical direction and coding standards, and reviewing developer or contractor work (required).
Strong Python and SQL proficiency, plus experience with C#/.NET or TypeScript (required).
Demonstrated experience shipping LLM-powered applications to production users, including tool use/agents, retrieval, structured outputs, and prompt/context design (required).
Experience evaluating AI systems by designing test cases and regression suites, diagnosing failures, tuning against measured results, and validating outputs against source data (required).
Hands-on experience with a cloud data platform and modern transformation tooling (examples: Snowflake, Databricks, Azure Synapse with dbt or Spark); Snowflake and dbt strongly preferred (required).
Experience with AWS or Azure, CI/CD (example: GitHub Actions) and infrastructure as code (Terraform) (required). AWS experience (ECS, Lambda, S3) is a plus.
Experience with Model Context Protocol (MCP), agent frameworks, or building tools and integrations for AI assistants (preferred).
Knowledge of data architecture including dimensional and semantic modeling, layered architectures, metadata/lineage, and data quality/reconciliation (required).
Understanding of security for data and AI systems: OAuth/OIDC, role/entitlement-based access to sensitive financial data, and auditability (required).
Production support experience (incident management, root cause analysis, remediation) (preferred).
Familiarity with private equity, asset management, transaction advisory or other financial services data (strongly preferred).
Experience with Power BI or similar reporting tools (preferred).
Hands-on use of AI coding assistants (examples: Claude Code, Cursor, GitHub Copilot) in a team setting, including code review and guardrails for AI-generated work (required).
Strong leadership, mentoring and communication skills, including turning ambiguous needs into working software and collaborating with business teams and contractors (required).

Technologies

Python, SQL, C#/.NET, TypeScript
LLM-powered applications; Model Context Protocol (MCP) and MCP servers
Snowflake, dbt, Databricks, Azure Synapse
AWS (ECS, Lambda, S3), Azure
CI/CD (GitHub Actions), Terraform, infrastructure as code
OAuth/OIDC, row-level security
Jira, Power BI
Claude Code, Cursor, GitHub Copilot
Unit tests, data reconciliation, AI evaluations

Benefits

Retirement benefits
Health insurance
Life insurance and disability
Paid time off
Paid holidays
Family planning benefits
Various wellness programs
Annual discretionary incentive program (eligibility)

Company profile

Global investment firm with $485 billion of assets under management across 678 investment vehicles as of June 30, 2026.
Founded in 1987 in Washington, DC.
Invests across Global Private Equity, Global Credit and Carlyle AlpInvest.

Position summary

Builds and runs a platform that brings together portfolio company, deal, fund, valuation and performance data from Carlyle’s investment, portfolio monitoring and enterprise systems.
Delivers data to investment teams, valuations, fund finance, investor relations and leadership through reporting, applications and AI-enabled tools.
Designs and ships production AI capabilities, including MCP services, agents and integrations for governed portfolio data on a cloud platform built on Snowflake, dbt and AWS.
Manager leads a team of contract engineers, writes code most weeks, sets the engineering and evaluation bar, and is accountable for accuracy, security and reliability of AI and data services.

Compensation

Compensation range is specific to Washington, D.C.
Comprehensive benefits package includes retirement benefits, health insurance, life insurance and disability, paid time off, paid holidays, family planning benefits and various wellness programs.

View full job page →

Get Llm Evaluations Engineer Jobs alerts

New Llm Evaluations Engineer Jobs, daily.