AI Training Scenarios - Reviewer (Quality Control)

About the Role

We're looking for highly analytical, detail-oriented professionals to serve as Quality Control Leads, focused on evaluating and auditing complex simulated environments and scenarios designed to train frontier AI agents. As an Auditor/Reviewer, you will be the final line of defense, ensuring that the digital worlds (spanning Gmail, Slack, Drive, etc.) and agentic tasks crafted by our design team meet the highest standards of realism, complexity, and technical accuracy. You will apply strict grading rubrics to assess scenario quality and provide vital feedback to the creators.

This is a W2 employment position with Cincinnatus LLC, with the opportunity to be placed at a leading AI Lab as part of their extended workforce. You will join a team of domain experts and together, you will guide the next generation of frontier AI tools and agents.

What You'll Do

• Audit & Evaluate: Review richly detailed personas, digital environments, and tasks created by AI Training Scenario Designers to ensure they are logically sound and properly structured.
• Apply Strict Rubrics: Utilize complex evaluation rubrics to score scenarios objectively, ensuring the tasks effectively challenge an AI agent's ability to reason, filter, and prioritize without being broken or impossible.
• Quality Assurance: Identify logical inconsistencies, factual errors, missing context, and technical flaws (e.g., invalid JSON structures) within the simulated environments.
• Provide Feedback: Deliver clear, constructive, and actionable feedback to the scenario designers to help them refine their worlds and improve their overall output.
• Document Outcomes: Maintain rigorous quality standards by logging your review outcomes, scores, and feedback clearly in Airtable and Crucible.

You're a Good Fit If You

• Have a proven track record in QA, editing, auditing, or reviewing complex written/technical content.
• Ability to engage for ~40 hours/week
• Excel at learning and rigorously applying comprehensive grading rubrics (prior high scores in rubric-based training or academies is a massive plus).
• Have an "eagle eye" for inconsistencies, edge cases, and logical gaps that others miss.
• Are comfortable giving constructive, structured feedback to peers and creators.
• Can quickly navigate structured tools like JSON editors, Airtable, and AI platforms. Must Have: Undergrad degree, 2+ years professional experience. Nice to Have:
• Editing / QA background — Senior editor, QA tester, technical reviewer, compliance auditor, or peer-reviewer.
• Technical / data comfort — JSON familiarity, Jira/Asana power user, data analyst, SQL/Python exposure. You don't need to write the code from scratch, but you must be able to spot when a JSON structure is broken.
• Process & Systems thinking — Operations managers, technical PMs, or systems designers who understand how complex, multi-stakeholder workflows should logically function in a professional setting.

About Cincinnatus LLC

Cincinnatus LLC is an enterprise staffing company that partners with leading technology companies to source and employ highly skilled professionals for contingent and contract-based opportunities. Cincinnatus serves as the employer of record for these engagements, providing W-2 employment, payroll, benefits, and compliance, while placing employees directly within client teams to work on high-impact initiatives.

Roles hired through Cincinnatus are not project-based or freelance engagements. They are structured, role-based positions that typically involve part-time or full-time commitments, close collaboration with a client's internal teams, and integration into standard enterprise workflows.

Cincinnatus is a legal entity separate from Mercor. While opportunities may be discovered through Mercor's platform, employment, onboarding, payroll, and benefits for these roles are administered by Cincinnatus LLC.

Equal Employment Opportunity

Cincinnatus is proud to be an Equal Employment Opportunity employer. We do not discriminate based upon race, religion, color, national origin, sex (including pregnancy, childbirth, reproductive health decisions, or related medical conditions), sexual orientation, gender identity, gender expression, age, status as a protected veteran, status as an individual with a disability, genetic information, political views or activity, or any other legally protected characteristic.

Cincinnatus is committed to providing reasonable accommodations for qualified individuals with disabilities and disabled veterans throughout the job application process.