The Reliability of an Epilepsy Treatment Clinical Decision Support System

From Clinfowiki
Jump to: navigation, search

Standridge S, Faist R, Pestian J, Glauser T, Ittenbach R. The Reliability of an Epilepsy Treatment Clinical Decision Support System. J Med Syst. 2014 Oct;38(10):119. [1]

Review 1

Introduction

It is seen that the treatment of epilepsy is a difficult endeavor for physicians because of the heterogeneous nature of the disease, lack of evidence based treatment protocols and physicians’ inadequate knowledge of the latest protocols and technologies. Thus, a Clinical Decision Support system is found to be very useful in supporting physicians and reducing ADE and improving healthcare delivery. The authors have developed such a knowledge based CDS system for epilepsy treatment and are now testing the reliability of the ET- CDS(a component of the CDS) , as a part of validation testing.

Development

The CDS system framework being tested is named “ Children’s Hospital Resource In Selecting Therapy Individualized Expert” (CHRISTINE). CHRISTINE provides an environment for developing and implementing different expert systems such as the multi-expert epilepsy treatment CDS (ET-CDS) system, which in turn comprises of two applications: an expert opinion collection system and a patient data system. On entering the patient details, the output provided is a selection of recommended treatment options based on the disease description, clinical evidence and drug drug interaction records.

Methods

Consistency of the recommendations of epilepsy treatment clinical decision support system for five pediatric epilepsy syndrome profiles was tested in three areas including the preferred anti epilepsy drug choice, the top three recommended choices, and the rank order of the three choices.

Results

The CDS system demonstrated a 100 % reliability on 15,000 executions carried out on five common pediatric epilepsy syndromes ( 100% match rates on treatment reports and system generated reports well as a match to the order of recommended treatment interventions).

Conclusion

The ET-CDS system is a reliable CDS system in the treatment of pediatric epilepsy.

Discussion

Reliability testing is the first step in determining the efficiency of CDS systems. The authors aim to carry out further, rigorous validity testing for the same. Also the current study had limitations like restricting treatment options to only three.

Comments

I would be interested in knowing how effective the system was in reducing ADEs. According to Velickovski et al, three features define the success of a CDS: it seamlessly integrates with workflow, it is delivered at a time and place of decision and it recommends action that is approachable by a provider [2] I believe a system that is 100% reliable should include all these features.

Others have subsequently reviewed this article.


Review 2 by Jonzy

Introduction

Epilepsy clinical decision support (CDS) is complex. Standridge et al. gives three prominent reasons why this complexity continues to increase:

  • a lack of evidence-based treatment guidelines,
  • the clinician’s growing deficit of current diagnostic and therapeutic knowledge,
  • the heterogeneity of the disease.

Standridge et al. developed a content-validated, computerized clinical decision support system (CDSS) for the treatment of epilepsy. The focus of the CDSS is to provide the best antiepilepsy treatments at a pediatric facility. Rigorous validation testing was required before implementation of the CDSS. Reliability was verified first. Consistency of the epilepsy treatment CDSS was measured in three areas according to Standridge et al. Consistency study areas included the preferred antiepilepsy drug, the top three recommended choices, and the rank order of the three choices. [3]

Development

Standridge et al. states, when developed properly, CDSS developed for therapy management have been found to facilitate evidence-based therapeutic interventions, reduce the incidence of harmful medical errors, and improve healthcare efficiency. The developed framework, named “CHRISTINE”, stands for Children’s Hospital Resource In Selecting Therapy Individualized Expert. CHRISTINE provides an environment for developing and implementing different expert systems according to Standridge et al. One such system, the multi-expert epilepsy treatment CDS (ET-CDS), is a system designed to facilitate treatment choices and options for clinicians treating children with epilepsy. ET-CDS is made up of two applications which work together and share a database. The first application is an expert opinion collection system. The second application is where the treatment report is generated. Standridge et al. contends this ET-CDS system will provide a non-expert clinician with an expert’s knowledge in managing the exceedingly diverse disease of pediatric epilepsy. The purpose of this study was to provide preliminary evidence for the reliability of the ET-CDS system.[3]

Methods

The reliability of the ET-CDS system in this study was conducted and evaluated using a three-step process. Using the already developed ET-CDS system, five pediatric epilepsy syndrome profiles were created (Table 1).[3]

Table 1 Five common pediatric epilepsy syndromes

  1. Childhood absence epilepsy
  2. Juvenile absence epilepsy
  3. Juvenile Myoclonic epilepsy
  4. Benign epilepsy with centrotemporal spikes
  5. Lennox-Gastaut syndrome[3]

Three reliability checks were performed.

  1. 1,000 tests performed on the consistency of the ET-CDS system in providing a single preferred treatment choice.[3]
  2. This step tests the consistency of the ET-CDS system in providing three appropriate treatment choices over 1,000 renditions.[3].
  3. This time, however, for a given report to be considered a match with the original report, all three treatment recommendations had to match exactly; that is, the three treatment recommendations produced by the system had to appear in the same order as in the most recent original report.[3]

This same three-step process was repeated for all five epilepsy syndromes. Care was taken to simulate and match five representative epilepsy syndromes often observed in clinical practice. In total, 15,000 executions were sought across five different epilepsy syndromes representing three different clinically relevant medical decision-making scenarios.[3]

Results

Stamdrodge et al. states, the ET-CDS system demonstrated 100 % reliability whether one treatment choice was requested or three treatment choices were requested by the clinician, using a series of 1,000 independently generated treatment reports for each of five different epilepsy syndromes. The match rate of 100 % between the original treatment report and all subsequent generated treatment reports held across all 15,000 computer-based executions, irrespective of whether the choice of treatment options was allowed to vary, or remain constant. They demonstrated 100 % reliability on 15,000 executions involving a three-step process on five different common pediatric epilepsy syndromes. [3]

Discussion

Standridge et al. states, tools can be disseminated on the worldwide web without much regulation or analysis, the scientific community must hold reliability and validity testing in high regard. A faulty clinical tool could lead to serious harm through improper recommendations on testing, diagnosis, and treatment. It is important to establish a replicable strategy to prove reliability of future updated versions as new evidence-based practices emerge. Ultimately, the reliability testing will ensure that future revisions of the ET-CDS system do not introduce unanticipated errors. [3]

Jonzy’s Comments

The methodical, scientific testing of CDSS for safety is necessary before any CDSS implementation. The study is a bit simplistic and somewhat anticipatory of its own success, but required. This study is also limited to epilepsy in the pediatric environment. Unfortunately, after writing this review, I found another had already reviewed it quit well. Please see there review as well. [4]

References

  1. Standridge S, Faist R, Pestian J, Glauser T, Ittenbach R. The Reliability of an Epilepsy Treatment Clinical Decision Support System. J Med Syst. 2014 Oct;38(10):119. http://link.springer.com/article/10.1007%2Fs10916-014-0119-9#
  2. http://www-ncbi-nlm-nih-gov.ezproxyhost.library.tmc.edu/pmc/articles/PMC4255917/
  3. 3.0 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 Standridge, S., Faist, R., Pestian, J., Glauser, T., & Ittenbach, R. (2014). The reliability of an epilepsy treatment clinical decision support system. Journal of Medical Systems, 38(10), 1-6. doi:10.1007/s10916-014-0119-9. http://link.springer.com.ezproxyhost.library.tmc.edu/article/10.1007/s10916-014-0119-9/fulltext.html
  4. The Reliability of an Epilepsy Treatment Clinical Decision Support System. http://clinfowiki.org/wiki/index.php/The_Reliability_of_an_Epilepsy_Treatment_Clinical_Decision_Support_System