EngineerJobs.io
← Back to all jobs

Job Description

Disney Entertainment and ESPN Product & Technology is seeking a Software Engineer II to support reliability-focused evaluation for operational AI and observability tooling. On the Reliability Insights team, you will connect real-world telemetry, AI agent interactions, and operational logs to measurable outcomes that explain model performance, drift, and incident mitigation impact.

This role is based in New York, NY (onsite). The salary range is USD 124,900 - 167,400 per year, with a baseline of 3+ years of relevant experience and a Bachelor’s degree in Computer Science, Engineering, or equivalent experience.

What you will do

  • Design, build, and maintain automated feedback pipelines that capture operational outcomes (confirmed incidents, false positives, dismissed alerts) and correlate them directly to AI model predictions and agent actions.
  • Create programmatic evaluation frameworks to monitor model precision, recall, and drift over time, with automated threshold alerts that support model tuning while reducing alert fatigue.
  • Build end-to-end data pipelines that connect AI outputs to incident lifecycles and quantify business impact using metrics such as Service to Incident correlation, notification success rates, and Mean Time to Mitigation (MTTM).
  • Develop and deploy scalable APIs, metrics layers, and single-pane model health dashboards that communicate AI ROI, operational health, and performance narratives to leadership and technical stakeholders.
  • Implement mechanisms to capture structured end-user feedback and route real-world usage signals back into model improvement cycles.

Key requirements

  • 3+ years of applicable experience in backend and data pipeline engineering, including building scalable APIs (for example, FastAPI or Flask) and metrics or insight generation platforms.
  • Strong data platform engineering experience with distributed processing tools such as PySpark, SQL, Pandas, Databricks, and Snowflake to process and analyze telemetry data.
  • Practical understanding of model evaluation concepts including precision, recall, model drift, and ground-truth labeling approaches.
  • Hands-on experience with modern development practices: GitHub for version control, Docker for containerization, and cloud-native deployments on AWS and EKS.
  • Proficiency with AI-assisted development tools such as Cursor and Claude Code to improve engineering velocity.
  • Analytical skills to turn complex telemetry and operational logs into reliable reliability metrics and business insights.
  • Strong collaboration and communication skills, including partnering with operational teams (such as SRE and DevOps) to turn raw data into actionable outputs.

Technologies

  • FastAPI, Flask, PySpark, SQL, Pandas, Databricks, Snowflake
  • GitHub, Docker, AWS, EKS
  • Cursor, Claude Code, Datadog, Grafana, Conviva
  • GPT-4, Claude

Preferred qualifications

  • Experience with observability, telemetry, and monitoring platforms such as Datadog, Grafana, and Conviva.
  • Familiarity with agentic workflows and foundation models (such as GPT-4 and Claude) and how to evaluate output accuracy.
  • Knowledge of incident management workflows and SRE metrics including MTTR, MTTD, and alert suppression.
  • Prior experience with data attribution, metrics aggregation, or financial or performance visibility tooling in high scale environments.

Similar Jobs