Computer Decision Support as a Source of Interpretation Error: The Case of Electrocardiograms

From Clinfowiki
Revision as of 06:19, 22 October 2006 by Theophrastus (Talk | contribs)

Jump to: navigation, search


Study Objective: to determine the effect that the computer interpretation (CI) of electrocardiograms (EKGs) has on the accuracy of resident (noncardiologist) physicians reading EKGs.

Methodology: randomized, controlled trial using a two-period crossover design with matched pairs of subjects randomly assigned to sequencing groups; interpretive accuracy of EKG findings were measured.

Study results: Without the CI, subjects interpreted 48.9% of the findings correctly. With the CI, the number of correctly interpreted EKGs rose to 55.4%.

When the CIs that agreed with the gold standard (Correct CIs) were not included, 53.1% of the findings were interpreted correctly, and when the correct CI was included, accuracy increased to 68.1%.

When computer advice that did not agree with the gold standard (Incorrect CI) was not provided to the subjects, 56.7% of findings were interpreted correctly. Accuracy dropped to 48.3% when the incorrect computer advice was provided.

Subjects erroneously agreed with the incorrect CI more often when it was presented with the EKG (67.7%) than when it was not (34.6%).

Conclusion: Computer decision support systems can improve the interpretive accuracy of internal medicine residents in reading EKGs. However, subjects were influenced significantly by incorrect advice, which tempers the overall usefulness of computer-generated advice in this and perhaps other areas.

Discussion: It is commonly felt that CI of EKGs is useful(the CI 'at the very least provides a second opinion which can be accepted or rejected by a physician'). It is not proven if this second opinion is a benefit to the physician; although two studies have examined how accurate physicians are in accepting correct computer generated advice, none have examined how physicians use incorrect computer advice.

In this study, subjects (30 internal medicine residents, PGY-2 and 3) were matched by year of postgraduate training and divided into two groups; they reviewed EKGs which were classified under four categories: (a) Correct, the correct finding was given by the CI, (b) Nonspecific, the CI mentioned the abnormality on the tracing but did not give the diagnosis, (c) Wrong by exclusion, the CI did not mention the abnormality at all, (d) Incorrect, the CI of the finding was incorrect in its final diagnosis.

Binary measures were used to score subjects, and the data was analysed using the generalized linear model procedure. Wald statistics were used to test the results against the null condition, and two-tailed 95% confidence intervals were calculated. (Review note: the statistical methodology of this study is felt to be good.)

This study suggests a divide between computer decision support and human decision making. Reliance on the computer decision making process by the human interpreter can lead to an overonfidence of the computer's ability to generate the right diagnosis; bad advice by the decision support software can be incorporated into patient management, potentially leaing to patientr harm (the sdudy did not address this specific issue, but laid the grounwork to show how incorrect advice might not be easily dismissed by the human decision maker).

VM 10/21/06