A remote, contract opportunity for experienced operators to review AI outputs at AuraOne Human Data. This role focuses on evaluating results across machine learning engineer specialist operations workflows, grading for workflow correctness, policy adherence, and stakeholder fit, while flagging operational risk and documenting the right next steps so the modeling team can train on them.
Responsibilities
- Evaluate AI outputs against established machine learning engineer specialist operations workflows, playbooks, and firm policy for ML Engineer tasks.
- Assess tone, escalation logic, and stakeholder fit using a structured rubric.
- Identify operational risks, missed escalations, and policy-adherence gaps, assigning severity tags as needed.
- Document the appropriate next steps so the modeling team can use them for training data generation.
- Resolve disputed treatments in line with published playbooks or firm guidance.
- Maintain reviewer quality scores through inter-rater calibration cycles.
Requirements
- Direct hands-on experience in machine learning engineer specialist operations on real teams performing ML Engineer work.
- Ability to apply multi-page rubrics consistently across large batches.
- Clear written reasoning that specifies the policy or workflow being applied.
- Strong attention to detail and the capacity to flag when a prompt itself is the root cause of an issue.
- Reliable async availability for at least 10 hours per week.
Why this role matters
The role centers ML engineer specialist operations in actual work scenarios. AuraOne leverages experienced operators to evaluate outputs with the judgment of a senior peer, ensuring workflow alignment, policy compliance, and the unwritten rules that determine whether tasks get completed.
Example tasks
- Grade a model response to a real ML engineer operations ticket and assess workflow, tone, and escalation.
- Flag a missed escalation with the correct severity tag and propose the next step.
- Adjudicate a disputed playbook call between reviewers using firm guidance.
- Audit a 25-row batch for rubric consistency and report drift to the program lead.
Nice to have
- Prior experience training, calibrating, or QA-ing operations teams.
- Familiarity with AI-assisted workflow tooling and its failure modes.
- Bilingual experience for cross-region operations.
Skills
- Operational review
- Policy adherence
- Workflow judgment
- Stakeholder communication
- Machine Learning Engineer specialist operations
Work model
Remote work eligible in the United States. Independent specialist contractor arrangement. Employment type: CONTRACTOR. Applicants must be authorized to work from the US.
Compensation
Hourly rate confirmed after the interview process.
Application process
Apply through AuraOne's specialist intake for role-specific routing and review. Final project scope, schedule, and contractor terms are confirmed before placement.
