JAVIS Chat: A Seamless Open-Source Multi-LLM/VLM Deployment System to Be Utilized in Single Computers and Hospital-Wide Systems with Real-Time User Feedback

The rapid advancement of large language models (LLMs) and vision-language models (VLMs) holds enormous promise across industries, including healthcare but hospitals face unique barriers, such as stringent privacy regulations, heterogeneous IT infrastructures, and limited customization. To address these challenges, we present the joint AI versatile implementation system chat (JAVIS chat), an open-source framework for deploying LLMs and VLMs within secure hospital networks. JAVIS features a modular architecture, real-time feedback mechanisms, customizable components, and scalable containerized workflows. It integrates Ray for distributed computing and vLLM for optimized model inference, delivering smooth scaling from single workstations to hospital-wide systems. JAVIS consistently demonstrates robust scalability and significantly reduces response times on legacy servers through Ray-managed multiple-instance models, operating seamlessly across diverse hardware configurations and enabling real-time departmental customization. By ensuring compliance with global data protection laws and operating solely within closed networks, JAVIS safeguards patient data while facilitating AI adoption in clinical workflows. This paradigm shift supports patient care and operational efficiency by bridging AI potential with clinical utility, with future developments including speech-to-text integration, further enhancing its versatility.