Hindcast Benchmark Tests LLM Forecasting Without Future Leaks

Hindcast replays resolved Polymarket markets with a fixed pre-event Reddit archive, testing whether LLMs can make forecasts from past information rather than retrieve outcomes.

TL;DR — Hindcast evaluates LLM forecasters by replaying resolved Polymarket markets as if the model were at a past cutoff date. It uses a frozen Reddit snapshot and permits only posts written before that date, reducing leakage from retrieval and training data. The benchmark scores models against both the final outcome and the market price at the cutoff.

The problem: forecasting tests can leak the future

Forecasters are often judged by backtesting: replay questions whose answers are now known, ask what probability the system would have assigned before resolution, and score those probabilities against the outcome.

Prediction markets such as Polymarket make this easier because they record a public price over time. A market price at a given date can be treated as a contemporaneous human forecast based on information available then.

LLMs complicate that setup. A retrieval system may surface reports written after the event, turning the task into a lookup. Newer models may also be trained on data published after the outcome, so a question that was once in the future may now be inside the model’s training data. In both cases, a benchmark can reward recall while claiming to measure foresight.

Full analysis is available via x402 micropayment.