Building better AI benchmarks: How many raters are enough?

Human disagreement makes AI benchmarks difficult to reproduce, yet evaluations often use only one to five raters per item and reduce their responses to a majority vote. The study introduces an (N, K) framework—balancing the number of items (N) against raters per item (K)—to determine how annotation budgets should be allocated. It concludes that the best balance depends on the evaluation goal: broad sampling for majority accuracy, but deeper rating for capturing nuanced human opinions.

The Breadth-versus-Depth Trade-off

• The “forest” strategy rates many items with few raters per item.
• The “tree” strategy rates fewer items with many raters per item.
• Historically, AI benchmarks have favored the forest approach, typically using one to five raters per example.
• This approach can miss both the overall distribution of opinions and meaningful disagreement among raters.

Simulating Annotation Budgets

• The researchers built a simulator using real-world subjective datasets, including toxicity, hate speech, safety, offensiveness, and job-related tweet classification.
• They varied:
  • Scale (N): 100 to 50,000 total items.
  • Crowd (K): 1 to 500 raters per item.
• Thousands of configurations were tested for statistical reliability, including whether model comparisons reached significance at p < 0.05.
• The simulator also examined messy conditions such as highly imbalanced categories and tasks with multiple labels.
• The simulator has been released as open source.

Why Three to Five Raters Are Often Insufficient

• Low-rater evaluations may fail to represent natural human disagreement.
• They provide too little depth to reveal nuanced opinions and too little breadth to establish a reliable overall picture.
• In many settings, more than 10 raters per item are needed to produce results that reflect the variation in human judgments.
• More ratings per item can make model comparisons more statistically reliable.

The Evaluation Metric Determines the Optimal Strategy

• Majority-vote accuracy

  • If the goal is to determine whether a model agrees with the majority of people, rating more items is generally more effective.
  • This favors the forest strategy.

• Opinion range and nuance

  • If the evaluation must distinguish between responses such as “yes,” “maybe,” and “no,” more raters per item are essential.
  • This favors the tree strategy because only repeated ratings reveal the full distribution of human opinions.

• There is no universally optimal number of items or raters; the correct allocation depends on what the benchmark is intended to measure.

Reproducibility Without Unlimited Budgets

• An appropriately chosen item-to-rater ratio can produce highly reproducible results with roughly 1,000 total annotations in some settings.
• Spending more money does not guarantee reliability if the budget is distributed poorly.
• The study’s framework is intended to help benchmark designers choose the allocation that best fits their metric and data characteristics.

Moving Beyond a Single Ground Truth

• Many AI evaluations assume that every example has one objectively correct label.
• This assumption becomes increasingly problematic for subjective tasks involving toxicity, harmful intent, ethics, safety, or social interaction.
• Preserving disagreement instead of collapsing it into a plurality label can make benchmarks more representative of real human judgment.
• The authors argue that understanding disagreement is as important as measuring consensus.

Benchmark designers should first decide whether they need majority accuracy or a detailed picture of human opinion, then allocate ratings accordingly. In subjective evaluations, using substantially more than five raters per item may be necessary for reliable and reproducible conclusions.