IRT Reliability in AI Benchmarks

A simulation study tests when item response theory supports AI benchmark claims—and when regime mismatch, estimator infeasibility, or unreliable inferences can distort rankings and item-level conclusions.

TL;DR — The paper stress-tests IRT for AI benchmarking. Across 18,000 simulated conditions, it finds that estimator feasibility and inference reliability depend on benchmark regime: classical estimators can become infeasible in large settings, while scalable estimators can yield unreliable ranking and item-level inferences for small or nonnormally distributed model sets. The abstract does not map those estimator categories to the four named methods.

Core problem

The article examines when item response theory, or IRT, is reliable for AI benchmark evaluation. The abstract says AI benchmarks increasingly use item-level statistical models to estimate model capabilities, rank systems, select informative examples, and diagnose benchmark quality.

The central problem is regime mismatch. Standard IRT estimation tools were originally developed for human testing, but AI benchmarks often have fewer evaluated models, far more items, and capability distributions that may be skewed, clustered, or multimodal.

The paper therefore asks whether IRT inferences remain reliable under AI benchmark conditions, especially for model rankings, predicted performance, and item characteristics.

Full analysis, extracted claims, numerical results, entity graph, FAQ, related work, applications, and BibTeX are available via x402 micropayment.