Think Alouds
Ideally, “deep understanding” is what teachers want for students. How do we know the extent to which students have achieved it? Experienced teachers know that standard methods for assessing what students know can conceal as much as they reveal. Papers and exams, for example, offer little help for figuring out why a student has recorded a wrong answer or struggled unsuccessfully with an assignment. Conventional assessments are also subject to problems of validity. Because they rely on students’ ability to articulate themselves in formal language, papers and exams tend to conflate understanding with fluency. In history, this is not a problem in many instances of grading. Narrative and argumentation are central to the construction of historical knowledge, making a fluent command of language and the conventions of liberal discourse important skills in their own right. But for the purposes of my study, I wanted to learn whether students’ mental habits were changing, apart from their ability to express the products of their new cognitive skills. Sometimes, especially with first-year students, the tongue-tied harbor understandings that are covered up by poor performances on papers. The reverse is true, as well; sometimes articulate students are able to say more than they really understand. “The thorniest problem” of assessment, according to Wiggins and McTighe (1998), is differentiating between the quality of an insight and the quality of how the insight is expressed.
To deal with this problem, I decided to study how my uncoverage survey affected student thinking patterns using a technique called the think aloud protocol assessment. Think alouds seemed to offer a promising method for uncovering what conventional assessment methods sometimes miss: hidden levels of student insight and/or misunderstanding.
Think alouds are a research tool originally developed by cognitive psychologists for the purpose of studying how people solve problems. The basic idea behind a think aloud is that if a subject can be trained to verbalize his or her thoughts while completing a defined task, then the introspections can be recorded and analyzed by researchers to determine what cognitive processes were employed to deal with the problem. In fields such as reading comprehension, composition, mathematics, chemistry, and history, think alouds have been used to identify what constitutes “expert knowledge” as compared to the thinking processes of nonexperts. In history, Sam Wineburg (2001) has used think alouds to identify the “epistemological pillars” characterizing the mental world of professional historians, ways of thinking that are largely absent from the minds of high school history students. But to my knowledge, no one had ever used think alouds in history to assess learning gains for individuals over time.
My procedure was as follows. In the spring and fall terms of 2000, I selected six students to participate in a think aloud study from the thirty students in each section of my HI 132 survey. The twelve total students selected for the two think aloud studies represented a cross-section of students in terms of gender, grade point average, and major/nonmajors. For their participation, subjects were paid $10 an hour.
In week one of the course, I sat down with each student one at a time in a room equipped with a tape recorder. After training subjects how to verbalize their thoughts aloud, I presented them with ten assorted short documents concerning the Battle of the Little Bighorn, a subject most knew little about. Then I asked the student to think aloud while “making sense” of the documents before them. This was essentially the same task they would be asked to perform eight times over the length of the course on every History Workshop day, though in this case their thoughts would not be filtered by the task of composing an essay. With the tape recorder running, subjects read through the documents aloud, verbalizing any and all thoughts that occurred to them as they made sense out of the documents. When subjects fell silent, I would prompt them to elaborate on their thoughts at that moment.
The think aloud sessions lasted anywhere from 40 to 90 minutes. Some students made an attempt to read all the documents; others read selectively as they were free to do.
Ten weeks later, in the week following the completion of the term, I sat down again with the six subjects in each study and ran them through a second think aloud. As with the first think aloud, students were shown documents about a historical event that occurred outside the period studied in the course (in this case, the Haymarket bombing). When these think alouds were completed, I had tape recordings of two think alouds for each student, one pre-course and one post-course. At that point, I hired student researchers to transcribe the tape recordings for later analysis.
The next step in the process required coding the transcripts. To prevent my bias as the course designer from influencing these results, I contracted with an outside analyst to help with the task. In the spring term think aloud study, my partner was Dr. Tim Hall of Central Michigan University. Dr. Hall seemed like a good choice for an outside analyst because, while he taught his surveys at the time using conventional coverage methods, he indicated that he was open-minded about alternative approaches. In the second fall term think aloud study, I employed two senior history majors to assist with the project as independent evaluators.
When each of the evaluators had finished independently coding and rating the transcripts, we then compared our coded transcripts until reaching consensus on how to rate the students’ abilities in the six key cognitive habits taught in the course.
When the students’ abilities had been rated, we compared the first and second think alouds for each individual to determine whether students had made gains in their understanding of what it means to “think historically.”
One or two of the twelve students in the studies made dramatic gains in all categories, such as Student #7:
Student #7 |
TA #1 |
TA #2 |
change |
Questioning |
2 |
5 |
+3 |
Connecting |
2 |
4 |
+2 |
Sourcing |
2 |
4 |
+2 |
Inferencing |
2 |
4 |
+2 |
Multiple Perspectives |
2 |
4 |
+2 |
Limits to Knowledge |
2 |
4 |
+2 |
More typical was Student #2, who made dramatic gains in a few categories, but remained at the same level of performance in others:
Student #2 |
TA #1 |
TA #2 |
change |
Questioning |
4 |
4 |
-- |
Connecting |
2 |
4 |
+2 |
Sourcing |
1 |
4 |
+3 |
Inferencing |
3 |
4 |
+1 |
Multiple Perspectives |
2 |
3 |
+1 |
Limits to Knowledge |
2 |
2 |
-- |
Putting all the data together and averaging the results, final totals from the two think aloud studies were as follows:
| Averaged Scores from All Students | TA #1 |
TA #2 |
change |
Questioning |
2.4 |
4.0 |
+1.6 |
Connecting |
2.6 |
3.6 |
+1.0 |
Sourcing |
2.5 |
3.5 |
+1.0 |
Inferencing |
2.5 |
3.7 |
+1.2 |
Multiple Perspectives |
2.7 |
3.4 |
+0.7 |
Limits to Knowledge |
2.8 |
3.2 |
+0.4 |
As it happened, one of the students in the first think aloud study never attended class on the History Workshop days. This is the second day in each unit of the course, the day when historical thinking skills are introduced and practiced. This particular student wrote no History Workshop papers, and therefore received no benefit from the recurring task of historical sense-making, no help in the form of feedback on his progress toward more sophisticated levels of cognition. In other respects he was a bright and motivated student, faithfully attending class on the first and third days of each unit. (Later the student explained to me that he suffered from a severe mental block about writing.) I could never have planned for a more ideal “control” student in my think aloud study!
When my research partner and I had finished coding and evaluating the transcripts for all six students in the study, I asked Dr. Hall if he could guess which student failed to receive any formal instruction in historical thinking apart from what he picked up on his own while listening to lectures on Day 3 of each unit and from reading the historians Zinn and Johnson. “It’s obvious,” Dr. Hall said. “It’s Student #1.” Which was absolutely correct. Here is Student #1’s think aloud record:
Student #1 |
TA #1 |
TA #2 |
change |
Questioning |
2 |
2 |
-- |
Connecting |
3 |
4 |
+1 |
Sourcing |
3 |
3 |
-- |
Inferencing |
2 |
3 |
+1 |
Multiple Perspectives |
3 |
4 |
+1 |
Limits to Knowledge |
2 |
2 |
-- |
Conclusions from the Think Aloud Studies
The think alouds opened a fascinating window into the thought patterns of students before and after the course. In all six categories, students on average made gains. When the data is disaggregated, students typically made dramatic gains of several levels in one or more categories, while remaining stable in others. The question I would put to defenders of the traditional model for teaching the survey is this: Can coverage surveys match these gains in historical thinking? I suspect rather strongly they cannot come close.
Overall, the think alouds revealed cognitive enhancements that were not as dramatic as students claimed in their self-reports, but much greater than indicated by using comparisons of early and late Workshop papers.
Why did the think alouds register more gains that what I could see in student papers? I asked students about this. The answer was always the same: it has to do with my grading system, in which at the end of the course students are asked to present for a grade what they think is their best paper of the term. As a result, very few students invest a lot of effort in the final Workshop paper. By the time the final paper is written, they have already written and re-written earlier papers to present for a grade.
Other surprises were more interesting. In the think aloud sessions, I could see that under-performing students struggled less with historical thinking than with reading itself. Moreover, in the second set of think alouds, my assistants and I noted that some of the best insights and meaning-making came from students who, in the gradebook, were steady “C” performers. For them, deep understandings seemed to evaporate when they tried to wrestle their thoughts to paper. This told us that we had work to do if we wanted to distinguish between assessing understanding and assessing students’ ability to communicate their understanding. For underperforming students, we discovered, the unexpected roadblocks to learning historical thinking are poor reading comprehension and a poor facility with prose writing.
Finally, I was struck by the difference between what the think alouds showed students were learning and what students said they were learning in their unprompted survey answers. In the think alouds, students made the largest gains in the category of asking good historical questions. But when asked to articulate what they learned in the course, only a few students said they had learned to ask questions. What does this mean?
I suspect it means that students in my course continue to assign a low value to the skill of asking questions, even though it is the skill with which they become most adept. All their lives they have been taught to think that the good student is the student who knows the answer to questions. Asking a question feels like admitting they are “not good” at history. If this is true, I may have identified another misconception students have about history, or about learning in general. Since 2000, I have stressed more and more in my course the critical importance of asking questions.
Think alouds are not perfect assessment instruments. But when used in combination with other methods, the advantage they offer is greater sensitivity to students’ struggles to formulate problem solving strategies, employ skills, and develop insights. Papers, exams, and ex post facto commentary by students are helpful in their own ways. But they make the process of understanding seem more orderly than it is, covering up the confusion, the disorientation, the mimicry of correct responses, and the lucky guesses—all of which are good to know about when assessing teaching and learning.