View source for Evaluation of rule effectiveness and positive predictive value of clinical rules in a Dutch clinical decision support system in daily hospital pharmacy practice

This is a review of the 2013 paper by Rommers et al.<ref name="rommers">Rommers, M. K., Zwaveling, J., Guchelaar, H.-J., & Teepe-Twiss, I. M. (2013). Evaluation of rule effectiveness and positive predictive value of clinical rules in a Dutch clinical decision support system in daily hospital pharmacy practice. Artificial Intelligence in Medicine, 59(1), 15–21. http://doi.org/10.1016/j.artmed.2013.04.001</ref>

== Abstract ==

===INTRODUCTION===
Our advanced [[CDS|clinical decision support (CDS)]] system, entitled 'adverse drug event alerting system' (ADEAS), is in daily use in our hospital pharmacy. It is used by hospital pharmacists to select patients at risk of possible adverse drug events (ADEs). The system retrieves data from several information systems, and uses clinical rules to select the patients at risk of ADEs. The clinical rules are all medication related and are formulated using seven risk categories.
===OBJECTIVE===
This studies objectives are to 1) evaluate the use of the CDS system ADEAS in daily hospital pharmacy practice, and 2) assess the rule effectiveness and positive predictive value (PPV) of the clinical rules incorporated in the system.
===SETTING===
Leiden University Medical Center, The Netherlands. All patients admitted on six different internal medicine and cardiology wards were included.
===MEASURES===
Outcome measures were total number of alerts, number of patients with alerts and the outcome of these alerts: whether the hospital pharmacist gave advice to prevent a possible ADE or not. Both overall rule effectiveness and PPV and rule effectiveness and PPV per clinical rule risk category were scored.
===STUDY DESIGN===
During a 5 month study period safety alerts were generated daily by means of ADEAS. All alerts were evaluated by a hospital pharmacist and if necessary, healthcare professionals were subsequently contacted and advice was given in order to prevent possible ADEs.
===RESULTS===
During the study period ADEAS generated 2650 safety alerts in 931 patients. In 270 alerts (10%) the hospital pharmacist contacted the physician or nurse and in 204 (76%) cases this led to an advice to prevent a possible ADE. The remaining 2380 alerts (90%) were scored as non-relevant. Most alerts were generated with clinical rules linking pharmacy and laboratory data (1685 alerts). The overall rule effectiveness was 0.10 and the overall PPV was 0.08. Combination of rule effectiveness and PPV was highest for clinical rules based upon the risk category "basic computerized physician order entry (CPOE) medication safety alerts fine-tuned to high risk patients" (rule efficiency=0.17; PPV=0.14).
===CONCLUSION===
ADEAS can effectively be used in daily hospital pharmacy practice to select patients at risk of potential ADEs, but to increase the benefits for routine patient care and to increase efficiency, both rule effectiveness and PPV for the clinical rules should be improved. Furthermore, clinical rules would have to be refined and restricted to those categories that are potentially most promising for clinical relevance, i.e. "clinical rules with a combination of pharmacy and laboratory data" and "clinical rules based upon the basic CPOE medication safety alerts fine-tuned to high risk patients".


== Background ==

The authors seek to evaluate the usefulness of a pharmacy alert CDS system in deployment in a teaching hospital in the Netherlands.  

== Methods ==

Over a 5 month period, patients admitted to one of six medical wards in the teaching hospital were monitored.  Daily safety alert profiles were generated on all inpatients.  Alerts were manually reviewed by pharmacists and stratified into insignificant, alerts that resulted in provider contact, and alerts with provider contact in which advice was delivered to the ordering provider. 

== Results ==

The system generated 2650 alerts in the study timeframe.   90% were judged insignificant.    204 of the 2650 alerts (7.4%) resulted in the pharmacist delivering prescribing advice to the provider.  In 128 of these alerts, the pharmacist's recommendation was followed.

== Conclusion ==

The authors show that the CDS is capable of generating clinically significant alerts that can modify potentially dangerous medication orders.  The high level of insignificant alerts is noted, and the authors suggest that the most important work in the future should be targeted towards using other data to improve the specificity of these clinical alerts.  

== Comments ==

I think this study gives a reasonable first-order look at what a basic CDS can do in an inpatient pharmacy setting.  A 90% rate of insignificant alerts is likely to result in rapid [[Alert fatigue|alert fatigue]], and would limit the clinical usefulness of this system.  Also, this is not a point-of-entry CDS, it runs as a background process, once per day.  I suspect that having an interactive CDS that fires at the point of order entry would have far more clinical utility. 

== Related Papers ==
[[Effect of computerized prescriber order entry on pharmacy: experience of one health system]]

==References==
<references/>
[[Category: Reviews]]
[[Category:HI5313-2015-FALL]]
[[Category: CPOE]]
[[Category: Adverse drug event]]