Research

I study whether AI models know when they are wrong, and how to tell. It decides when a chatbot should answer and when it should pass the question to a person.

Google Scholar profile

Explore the data

Results from the second paper, averaged over three severities of each image problem. Switch the signal to compare what the models know with what they say.

2B, full precision
2B, 4-bit
7B, 4-bit

No degradation

JPEG compression

Motion blur

Low light

Glare

Rotation

Low resolution

Guessing0.60.70.80.91.0

Score for spotting its own mistakes, from 0.5 (guessing) to 1.0 (always knows). Hover a dot for details.

arXiv preprint 2607.22034, 24 July 2026

Small Vision-Language Models Know When They Are Wrong But Cannot Say So: A Two-Model Study of Stated versus Internal Confidence Under Realistic Image Degradation

M M Asif Ferdous

Read on arXiv Code and data

Small vision-language models carry useful knowledge of their own mistakes in their token probabilities, even when the confidence they state in words is almost useless.

I tested two small open-weight models, Qwen2-VL-2B-Instruct and SmolVLM-Instruct, on images degraded by compression, camera shake and poor lighting, across 3,800 predictions.

The confidence the models state in words barely tracks whether they are right. Their internal token probability detects errors well, which makes it the better signal for deciding when a model should hand over to a person. Under severe low light, neither signal can be trusted.

Bar chart comparing error-detection AUROC for stated and internal confidence across image degradations — Figure from the paper.

arXiv preprint 2607.24440, 27 July 2026

Bigger or Cheaper? Scale and Quantization Effects on Uncertainty Signals in Vision-Language Models Under Image Degradation

M M Asif Ferdous

Read on arXiv Code and data

On a fixed memory budget, a larger model compressed to 4-bit beats a smaller full-precision model on both accuracy and knowing when it is wrong.

I compared three Qwen2-VL configurations that fit the same consumer hardware: a small model at full precision, the same model quantized to 4-bit, and a larger model quantized to the same footprint, across 5,700 predictions on six kinds of image degradation.

Model size made the internal uncertainty signal far stronger, with error-detection AUROC rising from about 0.80 at 2B to 0.98 at 7B. Quantization barely changed accuracy but weakened that signal. The practical advice: choose the bigger quantized model.

Chart showing internal-confidence AUROC rising from the 2B to the 7B model across degradations — Figure from the paper.