Software Engineer II - AI Performance

On the Reliability Insights team, you will design and build data and reporting pipelines that help evaluate and demonstrate the real-world performance, precision, and business impact of Disney’s operational AI and observability tools. The work focuses on turning telemetry, AI agent interactions, and operational logs into reliability metrics that support model drift tracking and incident mitigation.

Key Responsibilities

• Design, build, and maintain automated feedback pipelines that capture operational outcomes (confirmed incidents, false positives, dismissed alerts) and correlate them directly back to AI model predictions and agent actions.
• Develop programmatic evaluation frameworks to track model precision, recall, and drift over time, including automated threshold alerts to proactively trigger model tuning and reduce alert fatigue.
• Build end-to-end data pipelines connecting AI outputs to incident lifecycles, using metrics such as Service to Incident correlation, notification success rates, and Mean Time to Mitigation (MTTM) to quantify business impact.
• Design and deploy scalable APIs, metrics layers, and single-pane model health dashboards that communicate AI ROI, operational health, and performance narratives to leadership and technical stakeholders.
• Create mechanisms to capture structured end-user feedback and feed real-world usage signals back into ongoing model improvement cycles.

Requirements

• 3+ years of applicable experience in backend and data pipeline engineering, including building scalable APIs (FastAPI, Flask) and metrics or insight generation platforms.
• Strong experience with data platform engineering and distributed processing tools (PySpark, SQL, Pandas, Databricks, Snowflake) for processing and analyzing telemetry data.
• Practical knowledge of model evaluation concepts, including precision, recall, model drift, and ground-truth labeling systems.
• Hands-on experience with modern development practices: version control (GitHub), containerization (Docker), and cloud-native deployments (AWS, EKS).
• Proficiency with AI-assisted development tools (Cursor, Claude Code) to increase engineering velocity.
• Strong analytical skills to convert complex system telemetry and operational logs into reliability metrics and business insights.
• Excellent cross-functional collaboration and communication skills, including experience partnering with operational teams such as SRE and DevOps to turn raw data into actionable insights.

Technologies

• FastAPI, Flask
• PySpark, SQL, Pandas
• Databricks, Snowflake
• GitHub, Docker
• AWS, EKS
• Cursor, Claude Code

Additional Information

• The hiring range for this position in New York, NY is USD 124,900.00 to USD 167,400.00 per year.
• Base pay may vary based on internal equity and candidate’s geographic region, job-related knowledge, skills, and experience.
• A bonus and/or long-term incentive units may be included in the compensation package, and benefits may include medical and financial options depending on level and position.

Location and Education

• Location: New York, NY (onsite)
• Education: Bachelor’s degree in Computer Science, Engineering, or equivalent experience