A reproducible LLM-assisted grey-literature synthesis pipeline with human-in-the-loop validation

Background

Public-health evidence in Europe is fragmented across 38 jurisdictions and 30+ official languages: regulations, ministerial circulars and clinical guidelines sit in national legal systems rather than in indexed peer-reviewed literature, making cross-country comparison slow and error-prone. Large language models (LLMs) speed literature synthesis, however unverified outputs are unsafe for policy. We present a scope-agnostic pipeline coupling typed agentic synthesis with structured human-in-the-loop (HITL) validation.

Methods

During the 1stStage (Engine) a state machine decomposes a question into language- and locale-aware subgoals, retrieves evidence via web/API connectors from scrapable web content, and classifies sources under a typed schema. A three-tier LLM router drafts a cited report with adaptive verification where quantitative/regulatory claims are verified against source documents. Quality is tracked across replicated runs; disagreements are adjudicated by a prefilter–retrieval–controller protocol against fresh retrieval. Throughout the 2ndStage (HITL) human labelers work on a web-based interface that lets them rank, edit and highlight relevant information per subgoal, anchored against source PDF/HTML documents sources; this supports validation, evaluation, and downstream model improvement.

Results

An ongoing pilot on continuing medical education across 32 European jurisdictions and 5 healthcare professions ingested 216 field-level verdicts, 32 country-level comparisons, 960 profession-level ratings. A preliminary analysis of 125 1st stage verdicts was labeled by two reviewers: 39% confirmed, 37% enriched the record, 22% rewritten, 2% were flagged as uncertain based on reference data.

Conclusions

Coupling LLM synthesis with a HITL layer yields a transparent, reproducible methodology for grey-literature evidence mapping at inter-country scale, generalizable to systematic reviews and regulatory horizon-scanning.

Key messages

• A scope-agnostic pipeline coupling LLM synthesis with human-in-the-loop (HITL) validation that makes multi-country grey-literature mapping reproducible and audit-trailed.

• Preliminary results on a pilot study show LLM synthesis alone lacks in accuracy, thus requiring further validation and justifying the presence of a HITL layer.