Developing and evaluating an automated appendicitis risk stratification algorithm for pediatric patients in the emergency department

Abtract

"Objective

To evaluate a proposed natural language processing (NLP) and machine-learning based automated method to risk stratify abdominal pain patients by analyzing the content of the electronic health record (EHR).

Methods

We analyzed the EHRs of a random sample of 2100 pediatric emergency department (ED) patients with abdominal pain, including all with a final diagnosis of appendicitis. We developed an automated system to extract relevant elements from ED physician notes and lab values and to automatically assign a risk category for acute appendicitis (high, equivocal, or low), based on the Pediatric Appendicitis Score. We evaluated the performance of the system against a manually created gold standard (chart reviews by ED physicians) for recall, specificity, and precision.

Results

The system achieved an average F-measure of 0.867 (0.869 recall and 0.863 precision) for risk classification, which was comparable to physician experts. Recall/precision were 0.897/0.952 in the low-risk category, 0.855/0.886 in the high-risk category, and 0.854/0.766 in the equivocal-risk category. The information that the system required as input to achieve high F-measure was available within the first 4 h of the ED visit.

Conclusions

Automated appendicitis risk categorization based on EHR content, including information from clinical notes, shows comparable performance to physician chart reviewers as measured by their inter-annotator agreement and represents a promising new approach for computerized decision support to promote application of evidence-based medicine at the point of care."

Purpose

The authors of this paper chose to develop a method to triage abdominal pain pediatric patients into three tiers for possible appendicitis. This method used natural language processing (NLP) to gather data from the electronic health record (EHR) for analysis.

Materials and Methods

The researchers evaluated the pediatric patients by two methods. One method involved manually going through the records of the selected population, calculating the Pediatric Appendicitis Score (PAS) and then placing them into three groups (low-risk, equivocal-risk and high-risk) for appendicitis. The second method involved using an NLP algorithm developed by the team to mine the EHR for PAS elements that were then used to create a PAS. The researchers then used the computer developed PAS to assign the patients to the three above groups. Analysis was then used to compare the manual method to the computer generated method.