This evaluation benchmark pairs 161 line-image scans from a 1908 lithographed Urdu dictionary with native-verified ground truth text, alongside frontier-model error and confidence data.
Potential Use Cases
- OCR/VLM Benchmark Evaluation on Historical Script: Line images paired with human-adjudicated ground truth let OCR and vision-language models be scored directly against verified transcriptions of dense lithographed Urdu text.
- Model Error Pattern Analysis: Per-model word-level error and confidence-flag data expose where frontier OCR/VLM systems misread Nastaliq script, supporting targeted diagnosis of specific failure modes rather than an aggregate accuracy score alone.