Symbal Finds Recurring MLLM Caption Errors

Symbal introduces systematic misalignment detection: finding recurring MLLM caption errors tied to specific visual features in image-caption datasets.

TL;DR — Symbal detects systematic misalignments: recurring MLLM caption errors associated with specific visual features. The paper introduces SymbalBench, a benchmark with 1.7 million image-text pairs across 420 vision-language datasets, and reports 63.8% correct dataset-level identification, nearly 4x better than the closest unnamed baseline.

Problem and task

Symbal targets a specific captioning failure mode: repeated image-text errors that occur in association with a particular visual feature. The abstract calls these errors “systematic misalignments.”

The task setting is a vision-language dataset with MLLM-generated captions. The goal is to detect recurring caption mistakes tied to visual content. The abstract explicitly states that this auditing does not require access to the underlying captioning MLLM.

Full analysis, extracted claims, numerical results, entity graph, FAQ, related work, applications, and BibTeX are available via x402 micropayment.