Formative Evaluation of the Accuracy of a Clinical Decision Support System for Cervical Cancer Screening

These are reviews of Wagholikar et al's 2013 article, Formative Evaluation of the Accuracy of a Clinical Decision Support System for Cervical Cancer Screening.

First Review

Background

The authors of this article discussed the pre-launch testing performed on a clinical decision support (CDS) for cervical cancer screening. Instead of piloting the CDS in select patient care settings, they instead performed a formative evaluation wherein a side-by-side comparison was performed between interventions the CDS would recommend and what clinicians would recommend without the aid of CDS.

Methods

Clinicians who are target users of the cervical cancer screening CDS were asked to evaluate test cases, and select their recommendation from guideline-based options provided in the web interface. A total of 25 clinicians each evaluated 7 unique test cases. If their recommendations were different from what the CDS would have suggested, 2 clinicians who were not involved in the CDS development then reviewed the test cases.

Results

The clinicians and CDS matched in their recommendations in 94 cases. There were 75 mismatched recommendations, 22 of which were noted to be CDS failures due to modeling and programming errors. The rest of the mismatches were provider errors in recommending the best guideline-based order, which according to the authors would be optimized with the use of the CDS.

Conclusion

The authors concluded that the formative evaluation they used allowed them to pinpoint failures in the CDS before it was launched. They believe this method is better than pilot testing because they were able to test not only common patient scenarios, but also less frequent but very important cases.

Comments

I think the formative evaluation the authors used is highly effective. Aside from identifying CDS failures, it also enabled them to get a baseline data of how compliant clinicians are with guideline-based screening. However, I believe that pilot testing should still be done in conjunction with this evaluation see how the CDS performs - and how clinicians interact with it - in an actual patient care environment.

Second review

Background and Objective

The article discusses the importance of a prototypical clinical decision support system (CDSS), which screens for cervical cancer. Although screening is generally correlated with the prevention of cervical cancer, cervical cancer is continuously associated with a majority of cancer-related deaths in females. In fact, approximately 530,000 women were diagnosed with cervical cancer in 2008 and 275,000 deaths were associated with cervical cancer . Guidelines for the screening and surveillance of cervical cancer are complex in nature and fail to be recalled by health care providers. In a dire effort to maximize the optimization of cervical cancer screening, a prototype CDSS was developed to assess patient information in the electronic health record and suggest recommendations to care providers in accordance to specific guidelines. A meta-analysis comprised of a total of 42 studies conducted on a multinational scale reported that more than half of women diagnosed with cervical cancer either had no screening or inadequate screening. As more and more health care providers adopt the use of electronic health records in the United States, the use of CDSS as a mechanism to remind care providers and patients to adhere to screening guidelines will improve cervical cancer screening and surveillance rates.

Methods

Clinical decision support systems are developed to aid in prevention and diagnosis; however, ineffective implementation of CDSS impacts clinical outcomes due to fatigue, prolonged response time, and lack of accuracy and integration of workflow. The study utilized a web-based application to record recommendations of 89 potential end-users from a randomized sample of patients compared to those generated by CDSS. Recommendations that did not match between the random sample and CDSS were rectified by two experts in a dire effort to assess whether the provider recommendation or CDSS was more optimal. If the clinical decision support system proved to be less optimal than the provider recommendation, an error analysis was performed and the CDSS was improved to resolve errors accordingly.

Results

Twenty-eight of the 89 providers agreed to participate in the study; however, only 25 annotated a total of 175 test cases in accordance with their recommendations. Six of the cases were flagged with an error as the cases had bugs within the interface of the electronic health record system, which prevented providers from obtaining pathology reports. Of the remaining 169 cases, 75 cases had mismatched recommendations, of which 22 of those cases showed less than optimal CDSS. The error analysis led to the identification of 12 errors (modeling and programming errors) in the clinical decision support system. Analysis of research findings revealed areas within the guidelines that require decision support, especially among cases that demonstrate abnormal findings.

Comments

The use of CDSS to improve cervical cancer screening and surveillance has a unique set of potential advantages. However, CDSS also entails a subset of implications thereby requiring further research. The most significant criticism is the fact that the study involved only 25 providers who actually participated and annotated test cases. The study must therefore be performed on a larger scale to determine whether the results generated from future studies coincide and support data collected from this web-based application. Hence, verification from the CDSS recommendations is required to enhance the provision of feedback necessary in improving the optimization of the system. Based on prospective pilot studies, CDSS may potentially be implemented within clinical practice to improve clinical outcomes.