Evaluation and Design Methodologies
Evaluation Methods in Informatics
This is a collection of short descriptions of evaluation methods used in informatics research. The collection includes both qualitative and quantitative methods. The quantitative methods include pre-experimental, true experimental, and quasi-experimental designs.
- 1 Ethnography
- 2 Model for Evaluation of Health Information Technology (HIT)
- 3 Return on investment
- 4 Knowledge Management
- 5 Critical Incident Technique
- 6 Randomized controlled trial (RCT)
- 7 Focus Groups
- 8 Oral History
- 9 Cognitive Ethnography
- 10 Ethnomethodology
- 11 Grounded Theory
- 12 Interrupted Time Series Design with Comparison Group
- 13 Protocol Analysis
- 14 Video Analysis
- 15 Cost-Effectiveness Analysis
- 16 Workflow Analysis
- 17 User Centered Design
- 18 Articles Reviewed
- 19 Delphi method
- 20 References
Ethnography is the in-depth study of a group of individuals who share a common culture. Typically, the group is studied in their natural setting over a long period of time, often months or years, and the researcher becomes immersed in the daily activities of the group. The focus of ethnography is to study everyday behaviors, with the intent of identifying cultural norms, beliefs, social structures and other cultural patterns.
Model for Evaluation of Health Information Technology (HIT)
Return on investment
Critical Incident Technique
Critical Incident Technique (CIT) is a form of criterion sampling. [Patton, 2002] CIT relies upon interviews of subjects who fall into a defined category or categories and who carry out defined work tasks. Recurring interview topics are recorded as incidents, and the incidents are analyzed so to develop a psychological profile of the subjects. The goal of CIT research is to understand the weaknesses involved with a particular task and to provide solutions to resolve those weaknesses. [Wikipedia, 2007]
Randomized controlled trial (RCT)
A randomized controlled trial (RCT) is a quantitative research method often used in the healthcare setting.
focus groups are a form of qualitative research. They provide marketing managers, product managers, and market researchers with a great deal of helpful information. A focus group is a structured discussion in which a small group of people (usually 5-12), led by a trained facilitator, discuss their perceptions, opinions, attitudes, and experiences. These groups of people are part of a discussion of selected topics of interest in an informal setting which typically lasts about two hours. Participants are free to talk with other group members as the discussions are loosely structured and the moderator encourages the free flow of ideas. The moderator will be given an outline which will consist of a few specific questions prepared prior to the focus group. These questions will then start up open-ended discussions.
Oral history can be defined as “a method of gathering and preserving historical information through recorded interviews with participants in past events and ways of life.” (Oral History Assoc.) Linda Shope explains “oral history might be understood as a self-conscious, disciplined conversation between two people about some aspect of the past considered by them to be of historical significance and intentionally recorded for the record … oral history is, at its heart, a dialogue.” (historymatters.gmu.edu). Alan Nevins of Columbia University was the first to establish a disciplined and systematic methodology for collecting and preserving these conversations with an eye toward having them available for future research. In the last 25 years, there have been major projects to preserve the oral histories of those that witnessed the great and tragic events of the 20th century, including, for example, World War II, and the Holocaust. In an informatics use, LaVerda et al (2006) added relevant health questions to an interview format being used in an ongoing Veterans History Project. Audio-taped interviews conducted with individuals plus a focus group were coded and evaluated. Results included a determination preventive health practices acquired during military service during World War II were instilled as lifelong habits.
Cognitive ethnography is rooted in traditional ethnography but differs from it in a fundamental way. Whereas traditional ethnography is concerned with the meanings that members of a cultural group create, cognitive ethnography is concerned with how members create those meanings. Cognitive ethnography employs traditional ethnographic methods to build knowledge of a community of practice and then applies this knowledge to the micro-level analysis of specific episodes of activity. The principal aim of cognitive ethnography is to reveal how cognitive activities are accomplished in real-world settings. Cognitive ethnography is a particularly apt method for studying instruction in both formal and informal settings, such as that found in medical instruction - in the classroom or on the wards. Cognitive ethnography looks at process: at the moment-to-moment development of activity and its relation to socio-cultural (often institutional) processes unfolding on different time scales. Traditional ethnography describes knowledge; cognitive ethnography describes how knowledge is constructed and used [Williams, 2006].
Ethnomethodology is a sociological discipline which focuses on the ways in which people make sense of their world, display this understanding to others, and produce the mutually shared social order in which they live. It is distinct from traditional sociology, and does not seek to compete with it, or provide remedies for any of its practices. Furthermore, ethnomethodology is concerned with the"how" (the methods) by which that social order is produced, and shared (1,2). It seeks to describe the practices (the methods) these individuals use in their actual descriptions of those settings. Ethnomethology may ask, how do people perceive understand and explain the world in which they live? Specifically, with patient care, it may ask, are these perceptions and beliefs about the world changed when we become ill? Ethnomethodology is particularly concerned with cultural differences in explanations offered, especially the influence of social norms on the communication process. This cultural context is particularly significant when a patient tries to understand or the doctor tries to explain a condition, for example, a disease or newly diagnosed cancer.
Grounded theory is a qualitative research method which found its original application in the psychological-social sciences. Recently it has been employed in information technology research, and in evaluation of medical informatics systems and processes. It is particularly useful in investigations of organizations and organizational change, and the resulting social interactions (roles) of the players within those organizations. A basic premise of grounded theory is that it is indeed a theory discovery or development tool, “grounded” by observational or empirical data regarding the particular topic or phenomenon being investigated. This is a major difference between grounded theory and other qualitative methods.
Components of grounded theory include field observations by research or evaluation team members using a common reference frame, and both personal (individual) oral history interviews and focus group discussions. Field notes from observations and interview transcripts are then coded, sorted, compared and analyzed for relevant content. In the analysis phase, certain “themes” or categories pertaining to the topic become apparent; common traits or linkages discovered among themes or within themes are then identified and undergo further evaluation. Through a continuous, iterative process (data collection—coding—analysis), explanatory theory emerges.
For an example of the use of grounded theory in medical informatics see: Ash JS, Fournier L, Stavri PZ, Dykstra R. Principles for a successful computerized physician order entry implementation. AIMA 2003 Symposium Proceedings, pp. 36-40.
Interrupted Time Series Design with Comparison Group
Interrupted Time Series Design with Comparison Group, or ITS-CG, is one of a number of quasi-experimental research designs characterized by the absence of true randomization of the study and comparison groups. The ITS-CG method, like the related simple time series method, has multiple points of observation before and after an intervention or treatment. In addition, ITS-CG includes a non-equivalent (not randomly assigned) comparison group that did not undergo the intervention or treatment. The comparison group is chosen to be as similar as possible to the experimental group. Also, the repeated measurements should be equally spaced in time.
Protocol Analysis is a qualitative method that uses verbal descriptions of though processes and tasks to generate data about a given scenario or cognitive act. The primary technique used in protocol analysis is to ask subjects to "think-aloud" while performing a task. These verbal descriptions provide a set of explicit procedures that more thoroughly describe the solution to a task than other introspective techniques. According to K.A. Ericsson, "the central assumption of protocol analysis is that it possible to instruct subjects to verbalize their thoughts in a manner that doesn’t alter the sequence of thoughts mediating the completion of a task, and can therefore be accepted as valid data on thinking. Based on their theoretical analysis, Ericsson and Simon (1993) argued that the closest connection between thinking and verbal reports is found when subjects verbalize thoughts generated during task completion."
Video Analysis of systems performance involves videotaping all the interactions of a specified user population with the health information system (HIS) to determine whether HIS functions and components promote or inhibit more efficient or more effective patient care. Video analysis can be used in multiple ways; users may describe what they are doing and why as they work through a series of simulated or real tasks, or they may simply be taped as they go about their daily work. Consultants then analyze transcripts (generated manually or by computer) and actions observed on the tape to identify the functional and problematic components of the HIS. The recording logistics may be customized for each facility based on health unit work area configuration, user preferences, and financial resources. Using this method, consultants not only hear about HIS issues but also see what doesn't work. Furthermore, because consultants visually see problems, they need not ask users (who typically lack a software engineering background) how the system should be redesigned for better interactivity; they can use their education and experience to visualize solutions.
Cost-Effectiveness Analysis (CEA) seeks to quantify the benefits of a medical intervention through the use of a clinical outcome such as life expectancy, cost per life saved, or the number of adverse events (e.g., anaphylactic reaction) avoided. In CEA, the interventions being compared must be measured using the same measurement of cost (dollars) and the same measurement of outcome (a clinical outcome). The standard metric is the incremental cost-effectiveness ratio (ICER), the ratio of net costs to net benefits in dollars per unit outcome. Expressed as an equation by Friedman and Wyatt (1),
ICER = CostB – CostA / EffectivenessB – EffectivenessA
In medical informatics, CEA involves such comparisons as the cost of system (e.g., drug alert) implementation with the cost of treating the adverse events that occur in the absence of the system.
User Centered Design
- Computational analysis of non-adherence and non-attendance using the text of narrative physician notes in the electronic medical record.
- Patient Accessible Electronic Health Records: Exploring Recommendations for Successful Implementation Strategies
- Visions and strategies to improve evaluation of health information systems: Reflections and lessons based on the HIS-EVAL workshop in Innsbruck
see Delphi method
- Sittig, D. F., & Singh, H. (2010). A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Quality and Safety in Health Care, 19(Suppl 3), i68-i74. doi:10.1136/qshc.2010.042085. http://qualitysafety.bmj.com/content/19/Suppl_3/i68.short