Difference between revisions of "Return on investment"

From Clinfowiki
Jump to: navigation, search
(Principal Use)
Line 24: Line 24:
  
 
All potential variables can be difficult to enter into the analysis. Return on investment analysis can be difficult to include all necessary variables and can not predict potential new variables, or how the investment may trigger unintended consequences. If the return on investment analysis is performed for a small pilot project, the analysis doesn’t always hold true for larger projects.
 
All potential variables can be difficult to enter into the analysis. Return on investment analysis can be difficult to include all necessary variables and can not predict potential new variables, or how the investment may trigger unintended consequences. If the return on investment analysis is performed for a small pilot project, the analysis doesn’t always hold true for larger projects.
 +
 +
== Evaluating success of EMR implementation ==
 +
 +
Dean Sittig, professor at UT Houston's School of Biomedical Informatics, has suggested a new set of criteria for determining  for an EMR implementation. Based on Koch's Postulates and Hill's criteria for causation, these criteria are designed specifically for EMR evaluation.
 +
 +
* Must have the hardware and software available before the effect is identified.
 +
** Need to at least estimate state of affairs before system is implemented…manual review
 +
* Show that clinicians are actually using the system that could produce the effect.
 +
* Show that the effect increases with increasing availability and usage of the system.
 +
* Show that all obvious “alternative explanations” for the effect are false.
 +
* Show the effect goes away when the system goes away.
 +
* Show that a similar effect occurs when a similar system is installed and used at a similar facility.
 +
  
 
==Examples in Informatics==
 
==Examples in Informatics==

Revision as of 18:55, 31 August 2015

Return on investment (ROI) analysis is a quantitative method of evaluation. It is used to assess the potential earnings outcome from potential investment or business project. ROI analysis can be used to assist in making business decisions about the future. The practice of using ROI to determine whether or not to move ahead with a project is called ROI Analysis. ROI is defined as the gain from an investment minus the cost of the investment all divided by the cost of the investment.

History

Return on investment analysis appears to have been in use for an extremely long time.

Principal use

Measuring the future gain from a given project or investment. In this case, the attempt to quantitatively assess the future performance of an IT project is the principal use.

One could approach the ROI from the perspective of the Institute of Medicine Report, Crossing the Quality Chasm

  1. Safe: Reducing adverse drug events, inappropriate testing
  2. Effective: Reducing drug costs through appropriate prescribing
  3. Efficient: Reducing drug, laborotory, or radiologic utilization
  4. Timely: Reducing wait times
  5. Patient-centered: Reducing length-of-stay while hospitalized
  6. Equitable: Provides data to demonstrate equal delivery

Advantages

Return on investment analysis is a performance measure which allows for a quantitative measure to be applied to a business case decision. This can help remove emotion from a decision by having a standard measure to compare one investment to another.

Shortcomings

All potential variables can be difficult to enter into the analysis. Return on investment analysis can be difficult to include all necessary variables and can not predict potential new variables, or how the investment may trigger unintended consequences. If the return on investment analysis is performed for a small pilot project, the analysis doesn’t always hold true for larger projects.

Evaluating success of EMR implementation

Dean Sittig, professor at UT Houston's School of Biomedical Informatics, has suggested a new set of criteria for determining for an EMR implementation. Based on Koch's Postulates and Hill's criteria for causation, these criteria are designed specifically for EMR evaluation.

  • Must have the hardware and software available before the effect is identified.
    • Need to at least estimate state of affairs before system is implemented…manual review
  • Show that clinicians are actually using the system that could produce the effect.
  • Show that the effect increases with increasing availability and usage of the system.
  • Show that all obvious “alternative explanations” for the effect are false.
  • Show the effect goes away when the system goes away.
  • Show that a similar effect occurs when a similar system is installed and used at a similar facility.


Examples in Informatics

Return on Investment for a Computerized Physician Order Entry System Rainu Kaushal, Ashish K. Jha, Calvin Franz, John Glaser, Kanaka D. Shetty, Tonushree Jaggi, Blackford Middleton, Gilad J. Kuperman, Ramin Khorasani, Milenko Tanasijevic, David W. Bates Brigham and Women's Hospital CPOE Working Group J. Am. Med. Inform. Assoc. 2006;13(3):261-266. PrePrint published May 1, 2006; doi:10.1197/jamia.M1984

Comments on Return on Investment (ROI) As It Applies to Clinical Systems Mark E. Frisse J. Am. Med. Inform. Assoc. 2006;13(3):365-367. PrePrint published May 1, 2006; doi:10.1197/jamia.M2072 [Full Text] [PDF]

S. Wang A cost-benefit analysis of electronic medical records in primary care. The American Journal of Medicine, Volume 114, Issue 5, Pages 397-403

Return on Investment (ROI) Estimates

While barriers of determining actual ROI for EMR implementations exist, companies such as Dr. Cloud EMR are providing EMR and EHR ROI estimates based on each practice's details. This however does not suggest that it is entirely accurate and is only an estimate. DrCloudEMR is built by DrCloud Healthcare Solutions Inc, a wholly owned EnSoftek, Inc. subsidiary. [65] There are 2 main postulates for ROI which KOSH’s postulate and Sir Austin Bradford Hill’s criteria for Causation. Kosh’s postulate for CIS is i. The system or feature must be present in every case in which the benefit is observed. ii. The system must be isolated from the organization. iii. The benefit must be reproduced when the system is implemented in a new organization. iv. We must demonstrate that the system was used in the new organization. Hill’s Criteria for Causation includes (a) Strength of Association (b) Consistency of findings (c) Specificity of Association (d) Temporality (e) Dose-response (f) Plausibility (g) Coherence (h) Experimental Evidence and Analogy.

(a) Strength of Association tells us that the greater the change observed, the more likely the association is to be causal (e.g. If a EHR system is implemented and the CPOE feature greatly reduces medication errors, we could say that the implementation of the system had a causal effect on the reduction of medication errors and the strength of association is great).

(b) Consistency of Findings explains that if a change has been observed by different groups in different places with different circumstances and systems, the change is valid, so to speak. For example, if Company A (London, England, UK) implements System A , Company B (Houston, TX, USA) implements System B, and Company C (Guadalajara, Jalisco, Mexico) implements System C, and all three companies reduce medication errors using their respective systems, we can, again say that the CPOE feature of EHR systems can help reduce medication errors. It is important to note that the more consistent findings amongst different groups in different places, the better.

(c) Specificity of Association requires us to ask if there are any other factors which may have affected the change that we've observed. In regards to medication errors being reduced, one would have to ask if CPOE was the only factor involved. If errors could have been reduced due to other mechanisms in place besides CPOE alerts (e.g. better workflow in departments, new policies, etc.), the specificity of association could be considered weak. Weak does not imply wrong, but it does mean that more research has to be initiated.

(d) Temporality addresses the evaluation after an EHR system is implemented. Temporality asks us "were there any changes AFTER the system was implemented?" Usually this is harder to prove due to lack of data prior to EHR implementation, however, Sittig rates temporality as "strong."

(e) Dose-Response asks if the size of changes are directly correlated with the increase of system use (e.g. were medication errors greatly reduced due to the use of many medication alerts in the EHR system?). Usually, there is a strong and direct correlation between system use and the reduction of medication errors, as one example of a dose response in an EHR system.

(f) Plausibility must be shown; There must be some way to demonstrate that the EHR system was used the way it was intended to deliver certain results (e.g. Physicians must have used clinical support decisions the way the EHR system intended to reduce medication errors, in order to demonstrate plausibility.)

(g) Coherence simply states that changes caused by EHR systems should be caused by other EHR systems elsewhere. So, if medication errors are reduced by the use of one EHR system and that happens with the use of many other EHR systems, coherence exists.

(h) Experimental Evidence and Analogy is proving that when the system is not used properly or at all, that certain changes stop. So, if an EHR system is not being used properly or at all (after initial proper use), does a rise in medication errors resume? Experimental evidence is hard to obtain after EHR implementation because it requires not using the system for quite some time (which many would view as wasted money).



DESCRIPTION

Return on Investment (ROI) analysis is a quantitative approach that can be used to build a financial business case for any project. The term means that decision makers evaluate the investment by comparing the magnitude and timing of expected gains to the investment costs. Decision makers will also look for ways to improve ROI by reducing costs, increasing gains, or accelerating gains (1). Furthermore, ROI analysis can be used to make a decision about a future IT investment. For a prospective analysis, estimates of anticipated cost and performance are based on assumptions about the future (1,2). Another ROI approach involves a retrospective analysis that can show actual performance data about the IT project’s costs and returns (1). ROI analysis in general is a diverse collection of methods, skills, tools, activities, and ideas. They all may be useful for assessing the relative value over time of some investment. These methods are not, however, a single formula or predetermined calculation that will yield a simple yes-or-no answer to the question of how to invest. Consequently, a meaningful analysis of returns on investment in information technology is far easier said than done. Choices about how to conduct an ROI analysis should be based understandings about:

  • The strategic objective(s) of the analysis,
  • The place of the proposed IT investment in the overall project
  • How the analysis should be done (i.e., what data and methods of analysis are best suited to those objectives) (1,4).

HISTORY: As far as I can research this is technique has been used as long as costs are involved.

PRINICIPAL USE: Defining and measuring the costs and returns from IT investments. Stated another way: the measurement of the difference between the costs of and benefits from an investment (2).

ADVANTAGES: A methodology to help understand and assess the potential cost/investment needed for a project. While there may shortcomings as stated below, cost analysis must at least be done and ROI can provide a basic infrastructure.

SHORTCOMINGS: Measuring impact Assessing/analyzing scale/scope of project Often pilot projects do not reflect the true cost when the project is expanded on a larger scale Not much of an ROI literature base (3).

EXAMPLES IN INFORMATICS: 1. J Med Syst. 2006 Jun;30(3):159-68. Reviewing the benefits and costs of electronic health records and associated patient safety technologies. Describes the challenges in measuring return on investment (ROI) and reviews published ROI studies on health IT, including EHRs. Concluded that articles examining these benefits are much more common than studies examining ROI itself and additional research utilizing broader perspectives and multidisciplinary techniques will be needed before a better understanding of ROI from health IT is achieved.

2. J Assoc Acad Minor Phys. 2002 Jul;13(3):61-5. Return on investment analysis for a computer-based patient record in the outpatient clinic setting. The high cost of CPR implementation has been a major barrier to widespread acceptance of these systems. Describes a framework to evaluate the costs and benefits of implementing CPR systems in outpatient clinical settings. Return on investment (ROI), a, is one method to evaluate the economic implications of CPR. Supported the idea that understanding the ROI framework will enable physicians to make informed strategic decisions regarding purchase and implementation of CPR systems in their practices.

3. Am J Med. 2003 Apr 1;114(5):397-403. A cost-benefit analysis of electronic medical records in primary care. Electronic medical record systems improve the quality of patient care and decrease medical errors, but their financial effects have not been as well documented. The purpose of this study was to estimate the net financial benefit or cost of implementing electronic medical record systems in primary care. Implementation of an electronic medical record system in primary care can result in a positive financial return on investment to the health care organization. Electronic prescribing software is cost-effective for all size practices with a more rapid return on investment in larger practices.

  1. J Am Med Inform Assoc. 2006 May-Jun;13(3):261-6. 2006 Feb 24.

Return on investment for a computerized physician order entry system. Concluded hospitals may be able to save money and improve patient safety by investing in CPOE systems.

Challenges to ROI

However, calculating ROI is not a simple task. Issues that have hampered ROI studies and affected their validity include:

  • Pressure to justify expense
  • Shoddy collection of "before" comparison data after the implementation
  • Application of multiple simple statistical tests (the more statistical tests you run, the more likely you are to find something significant)

References

  1. Cresswell, A. Return on Investment In Information Technology: A Guide for Managers August 2004.
  2. J Assoc Acad Minor Phys. 2002 Jul;13(3):61-5.
  3. J Med Syst. 2006 Jun; 30(3):159-68.
  4. http://en.wikipedia.org

Amit Shah, MD As001