Evaluating health information exchange

From Clinfowiki
Revision as of 05:35, 15 October 2011 by Annathehybrid (Talk | contribs)

Jump to: navigation, search

Evaluating public health uses of health information exchange Jason S. Shapiro, Journal of Biomedical Informatics 40 (2007) S46-S49

Shapiro states that while the potential public health benefits of health information exchange (HIE) are clear there is little supporting evidence in the current literature. He notes that while there are several health use cases for HIE only a few have been implemented and even fewer have been evaluated.

The author lists six potential public health use cases for HIE (as described by Mostashari et al.)

  1. Mandated reporting of laboratory diagnoses: Shapiro contends mandated reports would be timely and complete if direct electronic reporting to state health departments via HIEs occurred. He proposes using the volume of relevant data (expressed as the ratio of relevant data reported to all relevant data in a laboratories database) as the primary evaluation measure. He also proposes pre- and post-implementation studies of efficiency, completeness and timeliness.
  2. Mandated reporting of physician-based diagnoses: Shapiro argues that HIEs would increase mandated physician reporting by automatic screening of data such as ICD-9 diagnostic codes, CPT codes and medications. However he acknowledges that the data so obtained would then have to be manually reviewed to exclude erroneous or non-relevant cases. He argues that with maturation of HIE systems more advanced informatics would lead to improved system recall. Unlike the first use case he does not see recall as a viable evaluation because of the cost and time constraints of determining all relevant data in the database. He advocates using precision (which would look at the number of erroneous or non-relevant cases that are included.)
  3. Public health investigation: Shapiro points out that in this use case the patient who has a reportable disease is already known and the health investigator need only access the HIE for additional information needed for the investigation rather than using more time consuming methods (making phone calls, travelling or obtaining paper records.) The ease with which this can be accomplished is dependent on the particular HIE (question of full electronic access vs. simple results retrieval.) Shapiro recommends using qualitative or semi-qualitative studies such as surveys, semi-structured interviews or observational techniques for evaluation.
  4. Disease based non-reportable laboratory data: HIEs will help centralize data and improve accessibility for health departments helping to guide public heath messages and rule out harmless epidemiological causes.
  5. Antibiotic-resistant organism surveillance: Culture resistant patterns would be transmitted through the HIE system to the health department and this information could then be used to construct community-wide antibiograms or notify clinicians about patients previously diagnosed with infections caused by an antibiotic resistant organism. Shapiro recommends evaluating rates of local resistance patterns as well as of nosocomial infections pre- and post- implementation.
  1. Population-level quality monitoring: HIEs would be used to monitor secondary and tertiary prevention of chronic diseases, e.g., rates of hemoglobin A1C levels for diabetes control. Quality metrics pre- and post- implementation could then be used to monitor preventive care.

Comments

Overall a good discussion of public health uses of HIE. However, I found approaches to evaluation in disease-based non-reportable data to be the least robust of all the use cases discussed. Also the evaluation of mandated reporting of physician-based diagnoses suggested by Shapiro involves manual review which would be time consuming.

Evaluating public health uses of health information exchange

Jason S. Shapiro




The potential of health information exchange (HIE) to improve the public health and public health activities may seem obvious, but current literature provides little evidence proving these effects. This article provides general evaluation methods for measuring the impact of HIE on public health in six use cases. For each use case the author gives a brief description for the case then discusses potential approaches to the evaluation of these use cases.


Mandated reporting of laboratory diagnoses.

  • Measuring the volume of relevant data transmitted using Electronic laboratory reporting directly to health departments would be a primary measure of success in this use case (i.e. recall): relevant ∩ retrieved/relevant. If feasible, a pre- and post-implementation study, could be conducted to demonstrate any change in the frequency of mandated reporting of laboratory diagnoses using HIE laboratory data.


Additional metrics of interest could look at changes in efficiency, completeness and timeliness of reporting.


Mandated reporting of physician-based diagnoses

Recall would be difficult to measure in this use case since measuring all relevant cases for the denominator would be time consuming and expensive. Another measure of interest is to evaluate the performance of the system (i.e. the precision): relevant ∩ retrieved/retrieved.

  • Measurement of precision would require a log file capturing all potential clinician reportable diseases detected by the system, followed by manual review to determine which cases are relevant.

Another useful evaluative measure might be to determine the volume of reporting pre- and post-implementation in much the same way that was suggested above for mandated reporting of laboratory diagnoses, as well as the efficiency, completeness and timeliness of reporting.

Public health investigation

Because implementation of this use case will vary depending on the extent to which the HIE incorporates clinical data, recommendation of a specific evaluation is difficult. One possibility is to do a qualitative or semi-qualitative study of the public health investigator’s experience through surveys, semi-structured interviews or observational techniques. In addition if this were done across multiple HIE projects, analysis might permit the development of a set of standard practices for HIE implementations to help identify the data elements and user interface features that are most essential to this use case.


Disease-based non-reportable laboratory data

Because this use case will rely on the development of faster, less expensive, and more accessible assays to clinicians, and these are being developed and deployed in parallel with HIE systems, they will likely act as a confounder, making it difficult to construct an evaluation plan that incorporates a pre-implementation phase.

Of most interest here is the gathering and analysis of empirical data to make new discoveries regarding the epidemiology of these common pathogens.


Antibiotic-resistant organism surveillance

Since the intervention here is the dissemination of a community-wide antibiogram, A study evaluating the rates of local antibiotic resistance patterns before and after the implementation of a community- wide antibiogram would be of interest. Similarly, studies could be done pre- and post-implementation on a hospital level to see if an antibiotic-resistant organism (ARO) notification system leads to earlier identification and isolation of ARO-infected patients, and to see if hospital rates of nosocomial ARO infections decrease.


Population-level quality monitoring

Again, this use case would require a pre- and post implementation study. A number of organizations have developed, or are developing, standard sets of quality measures that could be used for evaluation.


Conclusion

This paper describes preliminary suggestions for measuring the impact of HIE on public health in specific use cases. There are other secondary and tertiary benefits to improved public health that would be much more difficult to measure.

During the early phases of development of regional HIE, the projects will be limited in scope due to only partial penetration of local markets and therefore only partial data capture. Measures likely to be affected during this early phase are efficiency measures and costs. These early measures may be used to calculate a return on investment of the initial implementation costs.

As these HIE systems mature and begin to share data with one another, and a truly interoperable nationwide health information network (NHIN) begins to coalesce, quality and safety effects will begin to accrue and be measurable.


Limitations and Strengths

This article provides general evaluation methods for measuring the impact of HIE on public health in only six use cases and in two of this six cases , the author mention that it is difficult to construct an evaluation plan.

The good thing here is that he describe the use cases in some details and he also states some other benefits of the suggested evaluation measures in guiding and encouraging the process of HIE development and implementation.


by Ahmed Mahmoud


Evaluating public health uses of health information exchange Jason S. Shapiro

The potential of health information exchange (HIE) to improve the public health and public health activities may seem obvious, but current literature provides little evidence proving these effects. This article provides general evaluation methods for measuring the impact of HIE on public health in six use cases. For each use case the author gives a brief description for the case then discusses potential approaches to the evaluation of these use cases.

Mandated reporting of laboratory diagnoses

Measuring the volume of relevant data transmitted using Electronic laboratory reporting directly to health departments would be a primary measure of success in this use case (i.e. recall): relevant ∩ retrieved/ relevant If feasible, a pre- and post-implementation study, could be conducted to demonstrate any change in the frequency of mandated reporting of laboratory diagnoses using HIE laboratory data. Additional metrics of interest could look at changes in efficiency, completeness and timeliness of reporting.

Mandated reporting of physician-based diagnoses

Recall would be difficult to measure in this use case since measuring all relevant cases for the denominator would be time consuming and expensive. Another measure of interest is to evaluate the performance of the system (i.e. the precision): relevant ∩ retrieved /retrieved Measurement of precision would require a log file capturing all potential clinician reportable diseases detected by the system, followed by manual review to determine which cases are relevant. Another useful evaluative measure might be to determine the volume of reporting pre- and post-implementation in much the same way that was suggested above for mandated reporting of laboratory diagnoses, as well as the efficiency, completeness and timeliness of reporting.

Public health investigation

Because implementation of this use case will vary depending on the extent to which the HIE incorporates clinical data, recommendation of a specific evaluation is difficult. One possibility is to do a qualitative or semi-qualitative study of the public health investigator’s experience through surveys, semi-structured interviews or observational techniques. In addition if this were done across multiple HIE projects, analysis might permit the development of a set of standard practices for HIE implementations to help identify the data elements and user interface features that are most essential to this use case.

Disease-based non-reportable laboratory data

Because this use case will rely on the development of faster, less expensive, and more accessible assays to clinicians, and these are being developed and deployed in parallel with HIE systems, they will likely act as a confounder, making it difficult to construct an evaluation plan that incorporates a pre-implementation phase. Of most interest here is the gathering and analysis of empirical data to make new discoveries regarding the epidemiology of these common pathogens.

Antibiotic-resistant organism surveillance

Since the intervention here is the dissemination of a community-wide antibiogram, A study evaluating the rates of local antibiotic resistance patterns before and after the implementation of a community- wide antibiogram would be of interest. Similarly, studies could be done pre- and post-implementation on a hospital level to see if an antibiotic-resistant organism (ARO) notification system leads to earlier identification and isolation of ARO-infected patients, and to see if hospital rates of nosocomial ARO infections decrease.

Population-level quality monitoring

Again, this use case would require a pre- and post implementation study. A number of organizations have developed, or are developing, standard sets of quality measures that could be used for evaluation.

Conclusion

This paper describes preliminary suggestions for measuring the impact of HIE on public health in specific use cases. There are other secondary and tertiary benefits to improved public health that would be much more difficult to measure. During the early phases of development of regional HIE, the projects will be limited in scope due to only partial penetration of local markets and therefore only partial data capture. Measures likely to be affected during this early phase are efficiency measures and costs. These early measures may be used to calculate a return on investment of the initial implementation costs. As these HIE systems mature and begin to share data with one another, and a truly interoperable nationwide health information network (NHIN) begins to coalesce, quality and safety effects will begin to accrue and be measurable.

Limitations and Strengths

This article provides general evaluation methods for measuring the impact of HIE on public health in only six use cases and in two of this six cases , the author mention that it is difficult to construct an evaluation plan. The good thing here is that he describe the use cases in some details and he also states some other benefits of the suggested evaluation measures in guiding and encouraging the process of HIE development and implementation.

Reviewed by Ahmed Mahmoud


L. Bernard-Pantin