Evaluation and Design Methodologies

From Clinfowiki
Revision as of 14:28, 27 February 2007 by Ndgoldstein (Talk | contribs)

Jump to: navigation, search

Evaluation Methods in Informatics

This is a collection of short descriptions of evaluation methods used in informatics research.

Ethnography

Critical Incident Technique

Critical Incident Technique (CIT) is a form of criterion sampling. [Patton, 2002] CIT relies upon interviews of subjects who fall into a defined category or categories and who carry out defined work tasks. Recurring interview topics are recorded as incidents, and the incidents are analyzed so to develop a psychological profile of the subjects. The goal of CIT research is to understand the weaknesses involved with a particular task and to provide solutions to resolve those weaknesses. [Wikipedia, 2007]

Focus Groups

A focus group is a qualitative research technique where the researcher or moderator conducts a group interview to address a specific issue (1). The size of the group is variable, but in general, five to ten participants is considered optimal. The duration of the interview can be an hour or two. Focus groups are useful for several reasons (1,2,3). First, a group of interviewees can be brought together rapidly and cost-effectively. This is particular important for a health informatics project as continual user feedback during development and deployment is key to a projects success. Second, this informatics project impacts all users in the organization, and therefore having a common forum will help to encourage discussion and sharing of concerns on how the project affects the organization.

Oral History

Oral history can be defined as “a method of gathering and preserving historical information through recorded interviews with participants in past events and ways of life.” (Oral History Assoc.) Linda Shope explains “oral history might be understood as a self-conscious, disciplined conversation between two people about some aspect of the past considered by them to be of historical significance and intentionally recorded for the record … oral history is, at its heart, a dialogue.” (historymatters.gmu.edu). Alan Nevins of Columbia University was the first to establish a disciplined and systematic methodology for collecting and preserving these conversations with an eye toward having them available for future research. In the last 25 years, there have been major projects to preserve the oral histories of those that witnessed the great and tragic events of the 20th century, including, for example, World War II, and the Holocaust. In an informatics use, LaVerda et al (2006) added relevant health questions to an interview format being used in an ongoing Veterans History Project. Audio-taped interviews conducted with individuals plus a focus group were coded and evaluated. Results included a determination preventive health practices acquired during military service during World War II were instilled as lifelong habits.

Cognitive Ethnography

Cognitive ethnography is rooted in traditional ethnography but differs from it in a fundamental way. Whereas traditional ethnography is concerned with the meanings that members of a cultural group create, cognitive ethnography is concerned with how members create those meanings. Cognitive ethnography employs traditional ethnographic methods to build knowledge of a community of practice and then applies this knowledge to the micro-level analysis of specific episodes of activity. The principal aim of cognitive ethnography is to reveal how cognitive activities are accomplished in real-world settings. Cognitive ethnography is a particularly apt method for studying instruction in both formal and informal settings, such as that found in medical instruction - in the classroom or on the wards. Cognitive ethnography looks at process: at the moment-to-moment development of activity and its relation to socio-cultural (often institutional) processes unfolding on different time scales. Traditional ethnography describes knowledge; cognitive ethnography describes how knowledge is constructed and used [Williams, 2006].

Ethnomethodology

Ethnomethodology is a sociological discipline which focuses on the ways in which people make sense of their world, display this understanding to others, and produce the mutually shared social order in which they live. It is distinct from traditional sociology, and does not seek to compete with it, or provide remedies for any of its practices. Furthermore, ethnomethodology is concerned with the"how" (the methods) by which that social order is produced, and shared (1,2). It seeks to describe the practices (the methods) these individuals use in their actual descriptions of those settings. Ethnomethology may ask, how do people perceive understand and explain the world in which they live? Specifically, with patient care, it may ask, are these perceptions and beliefs about the world changed when we become ill? Ethnomethodology is particularly concerned with cultural differences in explanations offered, especially the influence of social norms on the communication process. This cultural context is particularly significant when a patient tries to understand or the doctor tries to explain a condition, for example, a disease or newly diagnosed cancer.

Grounded Theory

Grounded Theory is a qualitative research method which found its original application in the psychological-social sciences. Recently it has been employed in information technology research, and in evaluation of medical informatics systems and processes. It is particularly useful in investigations of organizations and organizational change, and the resulting social interactions (roles) of the players within those organizations. A basic premise of grounded theory is that it is indeed a theory discovery or development tool, “grounded” by observational or empirical data regarding the particular topic or phenomenon being investigated. This is a major difference between grounded theory and other qualitative methods.

Components of grounded theory include field observations by research or evaluation team members using a common reference frame, and both personal (individual) oral history interviews and focus group discussions. Field notes from observations and interview transcripts are then coded, sorted, compared and analyzed for relevant content. In the analysis phase, certain “themes” or categories pertaining to the topic become apparent; common traits or linkages discovered among themes or within themes are then identified and undergo further evaluation. Through a continuous, iterative process (data collection—coding—analysis), explanatory theory emerges.

For an example of the use of grounded theory in medical informatics see: Ash JS, Fournier L, Stavri PZ, Dykstra R. Principles for a successful computerized physician order entry implementation. AIMA 2003 Symposium Proceedings, pp. 36-40.

Protocol Analysis

Protocol Analysis uses verbal descriptions of though processes and tasks to generate data about a given scenario or cognitive act. The primary technique used in protocol analysis is to ask subjects to "think-aloud" while performing a task. These verbal descriptions provide a set of explicit procedures that more thoroughly describe the solution to a task than other introspective techniques. According to K.A. Ericsson, "the central assumption of protocol analysis is that it possible to instruct subjects to verbalize their thoughts in a manner that doesn’t alter the sequence of thoughts mediating the completion of a task, and can therefore be accepted as valid data on thinking. Based on their theoretical analysis, Ericsson and Simon (1993) argued that the closest connection between thinking and verbal reports is found when subjects verbalize thoughts generated during task completion."