Electronic medical records for clinical research: application to the identification of heart failure
Serguei Pakhomov, PhD; Susan A. Weston, MS; Steven J. Jacobsen, MD, PhD; Christopher G. Chute, MD, DrPH; Ryan Meverden, BS; and Véronique L. Roger, MD, MPH
Purpose and Background
The electronic medical record (EMR) is increasingly used in healthcare , it offers promising methods for identification of potential study participants, which is essential for clinical research beside its clinical goals . Indeed, although the use of manually coded patient records in clinical research is a long-standing tradition, it must allow for a delay between the diagnosis and the assignment of the code. In addition, coding systems have variable yields in identifying patients, depending on the disease.
Use of coding systems to identify patients appears particularly problematic for heart failure (HF) because of its syndromic nature, so the purpose of this article is to discuss and validate 2 methods of identifying HF through the EMR .
For this study, the authors used 2 data sources available as part of the Mayo Clinic EMR: clinical notes and diagnostic codes. then the authors began to discuss the first method which is the use of Natural Language Processing algorithm . The algorithm uses nonnegated terms indicative of HF , To maximize sensitivity, all available synonyms (n = 426) for these terms were used as well. NLP algorithm relies on finding nonnegated terms by excluding those terms that have negation indicators. the algorithm searched for the terms indicative of HF and their synonyms in the text of clinical notes as soon as the notes were dictated, transcribed, and became available electronically. Once a term was found, a determination with respect to its negation status was made . The algorithm was implemented in Perl programming language as an application that runs inside a JBoss Application Server. periodic verifications of the method to ensure that no patients with HF were being omitted by comparing the results of the algorithm with the billing codes. Predictive Modeling Algorithm based on the text of clinical notes involves an unlimited number of predictive covariates. algorithms specifically designed to process large numbers of covariates. Naïve Bayes is one such approach that is widely used in text classification. It has been shown to be functionally equivalent to logistic regression . This algorithm chooses the most likely outcome given a set of predictive covariates.
The outcome is dichotomous and the covariates are words found in the clinical notes . The likelihood of an outcome is computed based on its co-occurrence frequency with each of the predictive covariates. To extract covariates from text, The authors split the text of the clinical notes into single words listed in no particular order. They collected 2048 random clinical notes manually verified to contain evidence of HF (HF-positive examples) and 2048 random notes with no HF (HF-negative examples). Each note was then represented in terms of the vocabulary contained in all notes . The test set was created by combining one third of the HF-positive testing examples with two thirds of the HF-negative testing examples. The training set was created by combining 200 HF-positive examples with 600 HF-negative examples to force the predictive modeling algorithm to favor HF-negative cases and thus maximize the positive predictive value (PPV). The NLP and predictive modeling methods were evaluated using sensitivity, specificity, and PPV.
The NLP-based method provides 81.6% sensitivity (95% CI = 86, 100) and 97.8% specificity (95% CI = 97.7, 97.9), with a traditional diagnostic code based approach as a reference. As it identified The PPV was 49% (95% CI = 47.5, 51.1) using diagnostic codes as a reference and Of the 2904 patients identified by the NLP system as having active HF, record review showed that 1107 of these patients met HF criteria,18 leaving 1797 false positives and resulting in a PPV of 38% (95% CI = 36, 40). Of the 1472 cases identified by the NLP method but not by the diagnostic code method, 210 (14%) were manually confirmed to have active HF when Manual Review is the Reference. For Predictive Modeling the test set consisted of 66 HF-positive and 200 HF-negative examples. The naïve Bayes predictive model identified 37 of the 66 HF-positive examples and 192 of the 200 HF-negative examples correctly, yielding 56% sensitivity (95% CI = 44.1, 68.0), 96% specificity (95% CI = 93.3, 98.7), and a PPV of 82% (95% CI =73.1, 93.4). Natural language processing, yielded additional cases not captured by the billing system, thereby providing more comprehensive case identification. Predictive modeling, on the other hand, achieved a high PPV. A major advantage of either of these approaches over traditional methods that rely on diagnostic codes is that they enable case identification as soon as the text of a clinical note becomes available electronically, avoiding the delays and biases associated with manual coding.
Two methods, natural language processing and predictive modeling, were used to identify patients with heart failure from electronic medical records.
- Both approaches enable accurate and timely case identification as soon as the text of a clinical note becomes available electronically, avoiding the delays and biases associated with manual coding.
- Natural language processing may be more suitable for studies requiring the highest sensitivity such as observational studies.
- Because of its higher positive predictive value, the predictive-modeling approach is a better screening mechanism for clinical trials.
Limitations and Strengths
- The ICD-9–based billing system can be used only as an approximation of a criterion standard. The cases that were missed by both the billing system and the NLP system lie outside the range of the current evaluation.
- the results of this study may not be readily generalizable to other diagnoses; therefore, the use of EMR for patient identification has to be validated on a case-by-case basis .
- This study has unique strengths. It used a large dataset (more than 3000 patients) that was developed over a period of 3 years and involved complete manual records abstraction for validation.
- Another strength is that this study addressed identification of HF patients, whose diagnosis is complex, and relies in part on the language found in the unrestricted text of the EMR.
- NLP strategy may be used in EMR systems that do not routinely use problem-list entries.
- NLP approach is not restricted to a specific data element or a specific location in the EMR.