Skip to main content

AI Evaluation Rubric Auditor

Part Time$60 – $85/hrRemote · USA

The 4-day week in United States

About the role

Pay: $60-$85 per hour (USD).

The work

We are testing AI agents on realistic shopping and delivery tasks. Your role as a Project Auditor is to audit the work done by our Project Reviewers. You will audit complete tasks (prompt and grading rubric) and determine if their rubric failures are fair:

  • Was the information available?

  • Does the rubric accurately grade what the prompt asked for?

  • Would a valid answer have been marked wrong?

  • Is the rubric structure correct per the guidelines?

This role includes providing feedback to reviewers regarding prompt and rubric issues.

Must-haves

  • Substantial experience writing or reviewing rubrics for LLM evals. Please list the projects in your application.

  • Experience providing feedback to writers/reviewers

  • Consistent availability

  • Can read a long agent transcript and find where it went wrong.

  • Precise writing, and you check the data instead of trusting the reference answer.

About Mercor

Elite network connecting creative and technical experts with leading AI research labs for high-impact contract projects, offering transparent pay and flexible remote work.

See the full profile