Large language model accuracy in inhaler technique counselling for asthma and COPD: A comparison of free and paid models across ten devices
Abdurrahman Koç, Ferhat Sağun, Necmettin Öğe, Sami Avcil, Ali Tolga Çelik, Muhammet Ali Takeş, Bekir SunayObjective
To compare ten large language models (LLMs) from seven AI companies, across free and paid tiers, on inhaler technique instruction accuracy for ten devices, benchmarked against GINA 2025 and GOLD 2026 strategy reports.
Methods
In a cross-sectional, blinded evaluation, ten LLMs (ChatGPT Free/Plus, Claude Free/Pro, Gemini, Google AI Pro, DeepSeek, Microsoft Copilot, Perplexity AI, Meta AI) were queried between 10–17 March 2026 via each vendor’s official web interface. Fifty standardised prompts (10 devices × 5 question types) were submitted in triplicate on different days, yielding 1500 outputs. Two pulmonologists and one allergist scored seven metrics step-completion rate, critical and non-critical error counts, step-sequencing accuracy, safety-warning score, and Likert-scaled overall accuracy and patient comprehensibility against a gold standard derived from GINA 2025, GOLD 2026, the ERS/ISAM Task Force consensus, and manufacturer leaflets.
Results
Paid tiers outperformed free tiers on five of seven metrics (Mann–Whitney U; all
Conclusions
Paid subscriptions yield meaningful but non-uniform gains in inhaler-instruction accuracy. Safety-warning omissions and test-retest variability argue against autonomous LLM use; these tools are best deployed as adjuncts to clinician- and pharmacist-led teach-back education.