A Framework for Evaluating the Appropriateness of Clinical Decision Support Alerts and Responses

This is a review of McCoy et al's 2012 article, A Framework for Evaluating the Appropriateness of Clinical Decision Support Alerts and Responses.

Background

The authors noted that review of literature has shown that current methods of analyzing clinician responses to clinical decision support (CDS) alerts are not comprehensive. Their goal is to create a framework for evaluating the appropriateness or inappropriateness of alert overrides that could be used to evaluate other clinical decision support systems.

Methods

The authors developed an evaluation framework that considered both alert appropriateness measures and provider response appropriateness measures. The framework was then used to test a CDS for medication management of acute kidney injury (AKI) that the authors had previously launched. Their sample consisted of 300 patients that had AKI CDS alerts from November 2007 to October 2008. Two nephrologists analyzed the alert responses, with another nephrologist to adjudicate any subject matter disagreements.

Results

The authors noted that while only 20% of the CDS alerts were inappropriate, 82% of the alerts were overridden. But because this evaluation framework was set to measure clinician alert response not only during the immediate ordering timeframe, but tracked modifications for the alerted medication order even after 24 hours, the authors reported that clinicians later responded appropriately to CDS alerts by modifying or discontinuing the orders, which lowered the inappropriate clinician overrides to 17%. Expanding the timeframe of review for alert overrides considers the time that clinicians might need to verify their final decisions on medication orders with their peers, superiors, or published literature.

Conclusion

The authors concluded that their evaluation framework allows comprehensive assessment of CDS effectiveness, and can be modified by other institutions employing it depending on slight differences of the practice setting.

Comments

This article emphasizes the complexity of implementation and evaluation of effective clinical decision support systems. Clinical decision-making is not always linear; data gathering from electronic medical records does not always lend to the provision of the patient’s complete clinical picture, which is necessary for CDS systems to present the most appropriate alerts. Clinical decision supports will always need constant revision, but the use of evaluation frameworks such as the one suggested by the authors should allow for better identification of needed modifications.