This benchmark evaluates the text recognition performance of OCR and vision-language models (VLMs) using Arabic manuscripts and early printed books containing handwritten text and marginal annotations. It provides a held-out evaluation set separate from the training corpus, expert-verified ground-truth transcriptions, and per-model error analysis data. These resources allow developers to compare model outputs with verified transcriptions, assess recognition accuracy, and identify common recognition errors.
Potential Use Cases
- OCR/VLM Performance Evaluation on Arabic Handwriting and Marginal Annotations: Evaluate character- and word-level recognition accuracy by comparing model outputs with expert-verified transcriptions of complex Arabic documents, where existing models have shown word-recognition discrepancies of up to 96%.
- Cross-Model OCR/VLM Error Comparison: Use the provided per-model error analysis data to compare recognition errors across different OCR and VLM systems and identify weaknesses in recognizing handwritten text and marginal annotations.