Generative AI for Hospital Cybersecurity: A Framework for Evaluating Large Language Models for Planning, Threat Detection, and Incident Response

The increasing digitization of healthcare records and growing reliance on interconnected systems have increased hospitals’ exposure to cyber threats, including ransomware, brute-force attacks, and Structured Query Language (SQL) injection. Traditional cybersecurity approaches alone are often insufficient to address these threats, prompting growing interest in artificial intelligence (AI)-based decision-support tools. This paper evaluates the potential of ChatGPT for hospital cybersecurity and incident response while introducing a structured qualitative framework for evaluating Large Language Model (LLM)-generated cybersecurity recommendations in healthcare. Through three progressively designed experiments and a ransomware case study, we evaluate ChatGPT’s role in developing a hospital cybersecurity plan, detecting brute-force login attempts, responding to an SQL injection attack, and managing a ransomware incident. Responses are assessed using five evaluation dimensions: specificity, completeness, technical correctness, feasibility, and alignment with the National Institute of Standards and Technology (NIST) Cybersecurity Framework (CSF), including both explicit mapping and function coverage. The results demonstrate that ChatGPT provides structured, context-aware guidance that aligns well with NIST CSF 2.0 and addresses governance and third-party risks. However, the recommendations also exhibit limitations, including limited operational depth, assumptions about technology and regulatory environments, lack of prioritization for resource-constrained settings, and limited consideration of implementation costs. Overall, the proposed evaluation framework provides a systematic approach for assessing LLM-generated cybersecurity guidance, while the findings indicate that ChatGPT can serve as a valuable decision-support tool that should complement, rather than replace, qualified cybersecurity professionals.