A Framework for Evaluating the Appropriateness of Clinical Decision Support Alerts and Responses

This is a review of McCoy et al's 2012 article, A Framework for Evaluating the Appropriateness of Clinical Decision Support Alerts and Responses.

Background

The authors noted that review of literature shows that current methods of analyzing clinician responses to clinical decision support (CDS) alerts are not comprehensive. Their goal is to create a framework for evaluating the appropriateness or inappropriateness of alert overrides that could be used to evaluate other clinical decision support systems. A related study was done by McCoy AB, Thomas EJ, Krousel-Wood M and Sittig DF in summer 2014. The clinfowiki link to this article is Clinical decision support alert appropriateness: A review and proposal for improvement.

Methods

They developed an evaluation framework that considered both alert appropriateness measures and provider response appropriateness measures. The framework's effectiveness was tested by using it to evaluate a CDS for medication management of acute kidney injury (AKI)[1] that the authors had previously launched. Their sample consisted of 300 patients that had AKI CDS alerts from November 2007 to October 2008. Two nephrologists analyzed the alert responses, with another nephrologist to adjudicate any subject matter disagreements.

Results

The authors noted that while only 20% of the CDS alerts were inappropriate, 82% of the alerts were overridden. After further review using this evaluation framework, the percentage of inappropriate clinician overrides decreased to 17%. This is because the framework measured clinician alert responses not only during the immediate ordering timeframe; it also tracked modifications for the alerted medication order even after 24 hours. Some physicians performed delayed modification or discontinuation of the orders in response to the CDS alerts, with average delay time being 13 hours. Expanding the timeframe of review for alert overrides gives allowance for the time that clinicians take to verify their final decisions on medication orders with their peers, superiors, or published literature. Also, in a summer 2014 article named "Clinical decision support alert appropriateness: A review and proposal for improvement", the authors further mentioned some innovations to the alert evaluation methods which they believe would help reduce the alert overrides and improve patient outcomes.

Conclusion

The authors concluded that their evaluation framework allows comprehensive assessment of CDS effectiveness, and can be modified by other institutions employing it depending on differences in the practice setting.

Comments

This article emphasizes the complexity of implementation and evaluation of effective clinical decision support systems. Clinical decision-making is not always linear; data gathering from electronic medical records does not always lend to the provision of the patient’s complete clinical picture, which is necessary for CDS systems to present the most appropriate alerts. Clinical decision supports will always need constant revision, but the use of evaluation frameworks such as the one suggested by the authors should allow for better identification of needed modifications.

Related Article Review

Informatics Interchange - Alert Fatigue