Protocol Analysis (PA) uses verbal descriptions of thought processes and tasks to generate data about a given scenario or cognitive act. The primary technique used in protocol analysis is to ask subjects to "think-aloud" while performing a task. These verbal descriptions provide a set of explicit procedures that more thoroughly describe the solution to a task than other introspective techniques.
According to K.A. Ericsson, "the central assumption of protocol analysis is that it possible to instruct subjects to verbalize their thoughts in a manner that doesn’t alter the sequence of thoughts mediating the completion of a task, and can therefore be accepted as valid data on thinking. Based on their theoretical analysis, Ericsson and Simon (1993) argued that the closest connection between thinking and verbal reports is found when subjects verbalize thoughts generated during task completion."
The technique's greatest proponent is Dr. K. Anders Ericsson, from the Department of Psychology at Florida State University. Ericsson began using verbal reports of cognitive processes as early as his dissertation work in 1976. In the early 80's Eriksson and Simon wrote a seminal book on the topic: "Protocol Analysis: Verbal Reports as Data". Since then, the technique has become a major method in cognitive behavior research, thanks in part to the efforts of behaviorist John B. Watson, and gestalt psychologist Karl Duncker.
PA is widely used in usability testing, educational psychology, and interview/survey design. It has more recently been used as a corollary method to Discount Usability Engineering (usually with video taped recordings of the users actions instead of written notes) to allow rapid design and testing of user interfaces.
- PA allows researchers to gather data about cognitive processes "on the spot"; it does not rely on the subject's recollection of a process.
- When combined with video-taping, PA can be used to rapidly evaluate user-interfaces and to pinpoint areas of cognitive dissonance between the user's preferred workflow and the on-screen metaphor of the interface.
- PA can capture certain kinds of thinking (linear, workflow), but not all types of thinking.
- PA may not completely describe the process that the investigator is looking for, and may require secondary interviews or "probing". Ericsson and Simon recommend these secondary data be gathered with retrospective reports, so as not to interfere with the task itself.
- Because some subjects may find the process of thinking aloud to be uncomfortable or unnatural, they may actually alter their performance of a task during the process of PA.
Examples in Informatics
- Currie LM, Graham M, Allen M, Bakken S, Patel V, Cimino JJ. Clinical information needs in context: an observational study of clinicians while using a clinical information system. AMIA Annu Symp Proc. 2003;190-4.
- Gorman P, Lavelle M, Delcambre L, Maier D. Following experts at work in their own information spaces: using observational methods to develop tools for the digital library. J Am Soc Inf Sci Tech. 2002;53:1245-50.
- Kaplan B. Deriving design recommendations through discount usability engineering: ethnographic observation and thinking-aloud protocol in usability testing for computer-based teaching cases. AMIA Annu Symp Proc. 2003;346-50.
- Lilholt LH, Pedersen SS, Madsen I, Nielsen PH, Boye N, Andersen SK, Nohr C. Development of methods for usability evaluations of EHR systems. Stud Health Technol Inform. 2006;124:341-6.
- McKibbon KA, Fridsma DB. Effectiveness of clinician-selected electronic information resources for answering primary care physicians' information needs. J Am Med Inform Assoc. 2006 Nov-Dec;13(6):653-9.
- Johnson CM, Turley JP . The significance of cognitive modeling in building healthcare interfaces. Int J Med Inform. 2006 Feb;75(2):163-72.
- Jaspers MW, Steen T, van den Bos C, Geenen M. The think aloud method: a guide to user interface design. Int J Med Inform. 2004 Nov;73(11-12):781-95.
- Jaspers MW, Steen T, Van Den Bos C, Geenen M. The use of cognitive methods in analyzing clinicians' task behavior. Stud Health Technol Inform. 2002;93:25-31.
- Kushniruk AW, Patel VL, Cimino JJ. Usability testing in medical informatics: cognitive approaches to evaluation of information systems and user interfaces. Proc AMIA Annu Fall Symp. 1997;218-22.
--Desaib 12:31, 1 March 2007 (CST)