Design and Internal Validation of an AI‐Assisted Clinical Decision Support System (CDSS) for Telemedicine Services to Augment Patient Care in Rural Areas of Rajasthan

Background

Conventional teleconsultation platforms function primarily as communication interfaces and lack integrated clinical intelligence or adaptive learning capability. This study presents the development and design evaluation of the TeleMed Clinical Decision Support System (CDSS v2.5), an AI‐assisted predictive framework engineered to augment teleconsultation quality in rural areas of Rajasthan, India.

Methods

A development and internal validation design following TRIPOD‐AI guidelines was used to develop TeleMed CDSS v2.5. The framework integrated TabPFN 2.5 for tabular prediction, MiniLM embeddings for semantic symptom normalization, and KNN‐based clinical retrieval within an offline‐first, doctor‐in‐the‐loop telemedicine platform. Internal validation used stratified fivefold cross‐validation on the complete 4401‐record dataset. A clinician‐curated 100‐case edge‐case robustness suite, selected from the main dataset and reserved for secondary evaluation, was used to assess performance on linguistic edge cases and multidomain presentations. This suite is not a separate validation cohort to provide independent evidence of generalization. For comparative benchmarking, XGBoost, random forest, logistic regression, and a rule‐based triage model were evaluated using fivefold stratified cross‐validation on the same dataset. All AI‐generated outputs were reviewed, modified if necessary, and approved by the consulting clinician; only parent‐category classification was quantitatively evaluated.

Results

The TeleMed CDSS v2.5 achieved a macro AUROC of 0.962 (95% CI: 0.953–0.971) and accuracy of 87.2% (95% CI: 0.858–0.886) under fivefold stratified cross‐validation. On the clinician‐curated 100‐case edge‐case robustness suite, category‐level accuracy was 90.0% as an internal robustness assessment. Comparator model accuracies under fivefold cross‐validation were: TabPFN 2.5 standalone (0.831), XGBoost (0.845), random forest (0.827), logistic regression (0.792), and rule‐based triage (0.743). The complete multilayer pipeline improved upon standalone TabPFN by 10.7 percentage points on the 100‐case suite (90.0% vs. 79.3%). Contextual adaptation remained stable from the initial 2000‐record model through sequential expansion to 3000, 4000, and 4401 records, with accuracy differences from full retraining below 1 percentage point. The 8000‐record condition was a projected stress‐test and was not empirically observed during the study period.

Conclusions

The TeleMed CDSS v2.5 demonstrates technical feasibility of a hybrid AI framework integrating transformer‐based prediction, semantic symptom normalisation, and doctor‐guided contextual adaptation within a rural telemedicine workflow. These findings are limited to the design‐evaluation phase; external validation and prospective clinical effectiveness studies are required before broader deployment.

Trial Registration: CTRI: CTRI/2024/10/076070