May 2025 · Creator
Do AI models see charts like people do?
Thirteen image-capable models recognized a figure's purpose and dimensionality more reliably than its visual marks. Expert annotations across 305 scientific figures provided the comparison.
We showed the models figures and short category definitions, with captions and surrounding paper text withheld. This let us compare their labels with expert judgments about what was visible in the image.
Symbols, grids, and node-link diagrams caused the most errors. Models often chose a definite label even when unsure was available, and model size did not consistently predict agreement. Recognizing a figure’s general purpose left substantial gaps in recognizing how it encoded information.
With