Assessment of Clinical Decision Support Systems in the Diagnosis of Oral Radiographic Lesions

Abstract

ORAD III is a well-known system that produce a DD list of 10 forecasted diagnoses. It is a classical example of CDSS in oral radiology and diagnosis. We aimed to assess ORAD III in comparison to a rule-based system extracted from David MacDonald’s flowcharts. We used a structured form on Google Forms to gather radiographic features of 50 cases by two experts with 20 years and 15 years of experience, in a questionnaire modified from ORAD III 17 questions as well as MacDonald’s flowcharts. MacDonald’s flowcharts were programmed into an excel sheet. Data entry of both observers’ readings were fed into both systems and resulted in DD lists. The DD lists of the 50 patients were gathered and compared statistically to the gold standard diagnosis to measure accuracy of systems. Moreover, interobserver and inter-modality agreements of features recognition were performed. First ten diagnoses accuracy of ORAD III accuracy was 52.7% while that of MacDonald’s was 69.7%. First three diagnoses accuracy favored ORAD III 31.9% over MacD of 26.1%. Interobserver agreement and inter-modality agreement of radiographic features ranges from very good to weak agreement according to the feature itself, how many answers are different. The highest interobserver agreement was found with; jaw location, tooth displacement or impaction then root resorption (Kappa= 0.902, 0.592 & 0.587 respectively). The lowest interobserver agreement was found with; multilocularity, effect on pulp then trabecular pattern according to MacDonald’s terminology (Kappa= -0.218, -0.15 & -0.028 respectively).