Benchmarking End-to-End LLM-Based Systems for Document Understanding for Italian Newspaper Front Pages: A Pilot Study

This pilot study evaluates five LLM-based systems on newspaper front pages. Using ten front pages from Italian national daily newspapers, we assess 21 structured questions covering four capabilities: (i) bibliographic metadata extraction, (ii) layout understanding, (iii) localized information extraction, and (iv) visual semantic analysis. The evaluation includes plausible but unanswerable “trap” questions to assess hallucination and document grounding. The systems were evaluated through their official web interfaces. The benchmark therefore measures end-to-end document-understanding performance, including provider-specific document-processing pipelines, rather than the underlying LLMs alone. Because these configurations may differ in how textual, spatial, and visual information is extracted from uploaded documents, observed performance differences should not be interpreted as differences in the underlying models alone. Results show good performance on bibliographic extraction, while tasks requiring detailed layout, localized information extraction, and visual analysis are more challenging. Performance differences across systems may also partly reflect differences in their document-processing pipelines and interfaces. The evaluation framework, system outputs, and gold-standard annotations are publicly available to support reproducible research on document and layout understanding.