View source for Hazelhurst B, Sittig DF, Stevens VJ, Smith KS, Hollis JF, Vogt TM, Winickoff JP, et al. Natural language processing in the electronic medical record: assessing clinician adherence to tobacco treatment guidelines. Am J Prev Med. 2005 Dec; 29(5): 434-9

Automated Assessment of Adherence to Tobacco Treatment Guidelines

This study evaluates an automated system that abstracts data from the coded and free-text fields of the EMR to assess clinicians’ adherence to tobacco treatment guidelines. Even in state-of-the-art EMR systems, most data are entered as free-text narrative, and are therefore not amenable to currently available automated assessment methods. This study evaluated an automated EMR classifier (MediClass) that incorporates natural language processing techniques and handles both free-text and coded record data. 

This test of the MediClass system applied evidence-based guidelines for delivering smoking-cessation treatments in primary care settings. This study compared the presence or absence of each of the 5A’s of smoking cessation care as assessed by human abstractors and by MediClass. The 5A’s of smoking cessation care include:

·	Ask if the patient smokes

·	Advise the patient to stop

·	Assess the patient’s readiness to quit

·	Assist the patient in quitting by providing information or medications

·	Arrange for appropriate follow up care 

A preliminary study was run that allowed human abstractors to try out the entire process on an initial set of 500 records. Significant differences among abstractors were noted and follow up training was conducted. For the final study, the four trained medical record chart abstractors coded 500 records, each representing a single primary care visit. 

The human abstractors in this study had a difficult time consistently applying the 5A’s to all 500 records, despite careful training. MediClass agreed with the gold standard 91% if the time. Estimates of sensitivity were found to be 0.97, 0.68. 0.64, and 1.0, while for specificity, they were 0.95, 1.0, 0.96, and 0.82, respectively, across the four A’s (ask, assess, advise, and assist) for which measurement was possible.

This study demonstrates the feasibility of an automated coding system for processing the entire EMR, enabling automated assessment of smoking-cessation care delivery.

Comment: 
This is a well designed study. Significant effort was made in training the human abstractors. Even with this training, adjudication was still needed to establish a “gold standard” (thus it is more likely a silver standard). This coding task is difficult for people to agree on so I have to agree with the conclusion that MediClass is a practical alternative to expensive, inconsistent manual data abstraction.