Date of Award

1-1-2012

Document Type

Campus Access Dissertation

Department

Educational Studies

Sub-Department

Educational Psychology / Research

First Advisor

Richard E Clark

Second Advisor

Kellah Edens

Abstract

Cognitive task analysis (CTA) has evolved over the past half century to capture the mental decisions and analysis that experts have learned to implement when solving complex problems. Since expertise is largely automated and nonconscious, a variety of observation and interview strategies have been developed to identify the most critical cognitive elements of expertise so that they can be included in training and in machine learning.

While considerable evidence exists to support the learning and performance benefits of a number of CTA strategies, some of these strategies are much more time consuming and expensive to implement. The goal of this study is to test the cost-effectiveness of two different versions of a specific CTA strategy that has been tested successfully in many different experiments. The approach is called the ATI Cognitive Task Analysis System (ATI-CTA). In current version of ATI-CTA three experts receive a structured interview and each expert is asked to edit a summary of their interview. After all three experts have reviewed their summary; the three edited versions are combined into a "gold standard" version of the strategy used by the experts. In this experiment, a new approach was added whereby only one expert was interviewed and corrected their interview and then two additional experts reviewed and corrected the first expert's version.

The task chosen for this experiment was army recruiting. Six of the Army's top recruiters were randomly assigned to one of the two CTA interview strategies and separately interviewed on the subject of how to conduct the Army Interview for prospective recruits. Three subject-matter experts (SMEs) gave full ATI-CTA interviews that were transcribed and coded into a CTA report format. These three reports were then used to compile the gold standard (GS) for an approach that was designated the "3i+3r method." A fourth expert was given a full CTA interview and that experts report was then given to a fifth and sixth SME to review and correct. The final result was designated the "1i+3r GS." This method required significantly less time from the three experts. Time and cost tracking was carried out throughout the process and the number of decision steps elicited from each GS was calculated using two raters. Inter-rater reliability was calculated to be 93% in agreement for the 3i+3r method and 97% in agreement for the 1i+3r method. Results indicated that the 1i+3R method captured a greater number decisions steps (1i+3r = 35 and 3i+3r = 28), took less time (1i+3r = 26 hrs. and 3i+3r = 78.5 hrs.); and cost less (1i+3r = $1574.10 and 3i+3r = $5,235.55) than the 3i+3r method.

The conclusion of the study is that the less demanding 1i+3R method appeared to result in the identification of more decision steps in about 1/3 of the time and 30 percent of the cost of the 3i+3r method

Rights

© 2012, Catherine Flynn

Download

Share

COinS