AI-Generated Treatment Recommendations in Newly Diagnosed Lung Cancer: A Single-Center Feasibility Study in a Resource-Limited Setting

Background: How treatment recommendations generated by large language models (LLMs) compare with real-world decisions in oncology remains poorly characterized, particularly in resource-limited settings. Methods: In this single-center feasibility study, four LLMs (ChatGPT, Claude, Gemini, and Grok, accessed via consumer interfaces in June 2026) generated treatment recommendations for 27 consecutive newly diagnosed lung cancer cases treated at a Romanian oncology unit (June 2024–May 2026). Each model’s output and the treatment actually delivered were formatted identically (135 recommendations total), presented in randomized order, and scored for appropriateness (1–5) by a reviewer blinded to the source, with three quality flags. A second independent reviewer re-scored all recommendations to assess inter-rater reliability. Scores were compared using the Friedman test. Results: Mean appropriateness was highest for the real-world decision (4.52), followed by ChatGPT (4.48), Claude (4.44), Grok (4.41), and Gemini (4.37), with no significant difference (Friedman χ24 = 1.99, p = 0.74). No recommendation scored below 3. Inter-rater agreement was high in raw terms (93.3% within one point) but only fair after chance correction (weighted κ = 0.26), reflecting a ceiling effect. Conclusions: AI-generated recommendations were rated similarly to real-world decisions. The evaluation method proved feasible in a resource-limited setting; the small sample and ceiling effect preclude equivalence claims, and the findings are hypothesis-generating.