Classification models for the prediction of clinicians' information needs

From Clinfowiki
Jump to: navigation, search

This is a review for Guilherme Del Fiol and Peter J. Haug's Classification models for the prediction of clinicians’ information needs. [1]

Research question

Are classification models that are based on Infobutton usage data a promising solution for predicting what medication-related content topics (e.g., e.g., dose, adverse effects, drug interactions, patient education) clinicians are most likely to choose while entering medication orders in a particular clinical context?

Methods

Environment

A healthcare delivery network using a web-based EMR using Infobutton Manager.

Design

A dataset with 3,078 Infobutton sessions and 26 attributes describing characteristics of the user, the medication, and the patient. In these sessions was prepared with users selected one out of eight content topics. Redundant and useless attributes were elimanted using automatic attribute selection, from which a reduced dataset was used to produce nine classification models from a set of state-of-the-art machine learning algorithms.

Measurements

Classification performances by topic was measured by area under the ROC curve (AUC) and agreement (kappa) as predicted by the models and those chosen by clinicians in each Infobutton session.

Results

The models’ performance ranged from 0.49 to 0.56 (kappa). The AUC of the best model ranged from 0.73 to 0.99. Content topics adult dose, pediatric dose, patient education, and pregnancy category resulted in the best performance for choice prediction.

Main results

The five strongest individual predictors were avg reads, orders entered, avg writes, patient age, and parent level 3. Classifiers showed average kappa scores ranging between 0.47 (rules) and 0.56 (Stacking) indicating an overall moderate level of agreement. Stacking outperformed the other two competitors in all 10 bootstrapped test sets in spite of there being no statistical difference among the Stacking, Bayesian network, and SVM classifiers. The boosted algorithms were not statistically significance in their difference from their non-boosted counterparts. The learning methods showed varied performance levels regarding the prediction of each individual class.

Conclusion

The results suggest that while information needs are strongly affected by the characteristics of users, patients, and medications classification models based on Infobutton usage data are a promising method for the prediction of content topics that a clinician would choose to answer patient care questions while using an EMR system.

Commentary

In contemplating this study it is important to take particular note of the key limitation: because the attribute sets and classification models were developed specific to the researchers’ institution the specifics of the study will probably not generalize to other institutions. Recognizing that the outcomes are likely to be different on an institution by institution basis it is then possible to be guided by the learning methods and subset of attributes that were used in the study.

The overview of machine learning algorithms is helpful particularly when combined with the performance note that their experience was that Boosting algorithms did not significantly improve upon the performance on non-boosted counterparts contrary to prior studies, something that could be explained by the base algorithms that they chose to use were already strong learners rather than weaker ones used in previous comparison studies. Overall the researchers show strong awareness of the potential influences on the outcomes. For example they acknowledge that avg reads, avg writes, and orders entered are attributes that are more likely to appear in quantities for outpatient clinicians, potentially giving a better description of users than specialty and discipline.

They give a nice overview of some potential usability goals under Discussion: “to present the minimal amount of information to support quick decisions, reducing unnecessary navigation steps and exposure to irrelevant information.” That excellent summary would make an excellent goals statement for many infobutton or help button projects. The data transformation to Parent level 3, or main drug ingredient, was extremely well done.

The one slightly questionable area fell into data cleaning where sessions of more than four topics were categorized as likely to have been for “testing, demonstration, or training purposes” rather than confusion, misdirection or explorative hunting. Definitely an article worth reading for anyone looking to improve information retrieval via predictive classification models, as the methods are extremely interesting even if the results are likely to be different on an institution by institution basis.

References

  1. Guilherme Del Fiol, and Peter J. Haug. Classification models for the prediction of clinicians’ information needs. J Biomed Inform. 2009 February ; 42(1): 82–89. doi:10.1016/j.jbi.2008.07.00. http://www.ncbi.nlm.nih.gov/pubmed/18675380