P001 Real-world diagnostic performance of artificial intelligence in skin cancer screening

Abstract

Use of artificial intelligence (AI) in clinical settings is becoming commonplace. AI has recently been employed in the NHS skin cancer referral pathway. We sought to assess the real-world diagnostic performance of AI in this setting. The accuracy of the information from the AI algorithm passed to the subsequently reviewing clinician is important due to the risk of automation bias. This is a prospective observational study of the first 3 months of skin cancer referrals assessed by an AI algorithm in a tertiary care dermatology department in the North West of England. All lesions assessed by the algorithm were included in the analysis. Participant data were retrieved from medical records. Outcomes assessed included the AI diagnosis, whether a human review of AI diagnosis occurred, the face-to-face dermatologist’s diagnosis, and the outcome of the dermatologist’s assessment. Comparison was made particularly between the final histopathological diagnosis and AI and dermatologist diagnoses. AI had a sensitivity of 95.3%, which compared favourably with dermatologists (88.5%; P = 0.006). The positive predictive value of the AI algorithm was lower, at 46.5%. This compared with 62.1% in dermatologists. Overall, 318 AI assessments with no human review went on to have their lesions reviewed by a dermatologist and resulted in biopsy. AI correctly identified the precise diagnosis 28.6% of the time, compared with dermatologists 61.6% of the time (P < 0.001). The correct tumour/lesion type was identified by AI 51.4% of the time and by dermatologists 75.5% (P < 0.001). AI has high sensitivity in the detection of skin cancer. However, the diagnostic accuracy of the information provided by AI to clinicians is low and could be further optimized to reduce the risk of automation bias.