Comparing ChatGPT Assessments Based on Transcripts and Screenshots with Anesthesiologists’ Evaluations of Ultrasound-Guided Abdominal Wall Block Videos: A Cross-Sectional Study

Digital video-sharing platforms are widely used in anesthesia education, but the educational quality of ultrasound-guided abdominal wall block (UAWB) videos is heterogeneous. This cross-sectional study assessed agreement between a large language model (LLM) and experienced anesthesiologists in evaluating UAWB videos on YouTube. Seventy-five videos covering five UAWB techniques were independently evaluated by two anesthesiologists and ChatGPT version 5.2 using a researcher-developed 10-item Educational Content Checklist (ECC), DISCERN, the Global Quality Scale (GQS), and JAMA benchmark criteria. The LLM received each video’s full transcript, metadata screenshot, and 5–9 procedural screenshots rather than the full video. Human–LLM scores were compared using the Wilcoxon signed-rank test, and agreement was quantified using intraclass correlation coefficients. Total scores did not differ significantly for the ECC (median 34 vs. 32; p = 0.168), DISCERN (54 vs. 54; p = 0.347), GQS (4 vs. 4; p = 0.469), or JAMA criteria (2 vs. 2; p = 0.865). However, agreement was limited: moderate for ECC (ICC = 0.576, 95% CI 0.404–0.709) and JAMA (ICC = 0.529, 95% CI 0.343–0.674), and poor for GQS (ICC = 0.495, 95% CI 0.303–0.648) and DISCERN (ICC = 0.357, 95% CI 0.146–0.538). In exploratory item-level analyses, differences emerged in vascular safety, needle technique, and indications. The absence of significant differences in total scores should not be interpreted as equivalence. LLM-assisted pre-screening is a potential application that requires prospective evaluation, and the present findings do not support replacing expert assessment with LLM evaluation, particularly for dynamic, safety-critical technical content.