DigiPhrame AI: Automating Evidence-Based Evaluation of Digital Public Health Interventions

Background

Digital public health interventions are expanding faster than evaluation capacity. DigiPHrame spans 13 domains and 208 criteria, but manual application requires months of expert effort. We developed DigiPhrame AI, a web app that automates these assessments using large language models with hybrid retrieval-augmented generation.

Methods

DigiPhrame AI uses 13 domain-specific expert modules grounded in WHO, ISO, GDPR and WCAG to assess uploaded intervention documents and code. We conducted an automated meta-evaluation of 413 AI-generated criterion-level assessments across two interventions, BUMPER (cancer prevention app; n = 206) and MAMACARE (maternal health app; n = 207), using a custom 12-metric RAGAS framework tailored to structured outputs including score labels, evidence, justifications and recommendations.

Results

Overall quality was good (pooled composite 72.2%). Highest-performing dimensions were score label calibration (96.5%), evidence grounding (95.7%), assessment justification (92.8%) and response completeness (83.8%). Recommendation actionability (77.1%) and evidence sufficiency (75.5%) were also good. Median criterion-level quality was 71.2% (IQR 65.5–76.6; range 24.4–91.1%). Lower-performing dimensions were reasoning chain validity (57.7%), framework alignment (47.8%), criterion coverage (44.9%) and context relevance (40.6%), indicating priorities for optimization. BUMPER slightly outperformed MAMACARE on composite quality (73.4% vs 71.1%).

Conclusions

DigiPhrame AI shows promising initial performance for scalable, transparent evaluation of digital public health interventions while preserving evidence traceability and human oversight. These findings reflect an initial automated evaluation; human evaluation of AI-generated outputs is ongoing alongside continued RAG pipeline development. Open-source AI-assisted assessment could support researchers, developers and funders in making more systematic pre-deployment and quality-improvement decisions

Key messages

• DigiPhrame AI could strengthen digital health intervention evaluation by enabling faster, transparent and systematic assessment across 13 domains before real-world deployment.

• Across 413 automated assessments, DigiPhrame AI showed good overall quality (72.2%), with excellent score calibration (96.5%) and evidence grounding (95.7%).