koedinger@cmu.edu
Professor of Human Computer Interaction
and Psychology
Director of the Pittsburgh Science of Learning Center
Research Statement
Summer, 2004
My research
spans a number of areas within the fields of Cognitive Science, Cognitive
Psychology, Human-Computer Interaction, Artificial Intelligence, and Education.
Within Cognitive Psychology, my focus has been on the nature of complex human
problem solving and learning. I have
employed experimental methods and think aloud protocol studies to explore these
areas, but have especially used intelligent computer-based tutoring systems as
research vehicles to tightly control instructional manipulations, to run
learning studies of longer more realistic duration, and to collect detailed
micro-genetic data on the learning process.
Within Human-Computer Interaction and Education, I have coined the term
"expert blind spot" to summarize results of experiments showing that
domain experts can be systematically inaccurate in judging the difficulties of
novice users/students. Thus, designing
computer interfaces or instruction should not be based on intuition alone, but
should be informed by general psychological theory and domain specific
empirical data. I have developed methods
for efficiently collecting such data and for employing cognitive theory in the
design of Cognitive Tutors. I have run
laboratory and field experiments to demonstrate significant student learning
gains from Cognitive Tutors relative to alternative instructional methods. I have also helped to widely disseminate
these systems, so that, for instance, Cognitive Tutor Algebra is now in over
1700 schools across the
The first
sections of this statement are organized around my earlier pre-tenure research
goals:
1)
to make new
discoveries about the nature of complex problem solving and learning and
2)
to use these
discoveries in the design of intelligent tutoring systems and classroom
interventions that demonstrably increase student mathematics learning.
The final main
section focuses on my efforts, since tenure, to expand the cognitive modeling and
cognitive tutor research agendas. The
culmination of those efforts is the new Pittsburgh Science of Learning Center
that I will direct. The center is funded
by the National Science Foundation for 5 years and about $25 million and will
start in October, 2004.
My
basic research has investigated the role of informal, invented, or intuitive
knowledge in the learning of mathematical problem solving and reasoning. The well-structured and rule-bound surface of
mathematical reasoning gives the impression that the underlying thinking
process has the same structure and rules.
My basic research debunks this tempting impression and, furthermore,
illustrates what the rules of mathematical thinking are if they are not the rules of mathematics itself.
Research
on how people come to understand mathematical formalisms is not only directly
relevant to mathematics and science learning and instructional design, but it
is more generally relevant to the design of information technologies and their
human-computer interfaces.
In
my early research, I developed a cognitive theory of how good problem solvers
are able to make intuitive leaps of inference in discovering and creating
proofs of geometry theorems (Koedinger
& Anderson, 1990). A careful analysis of “think-aloud” reports of human
experts revealed that when planning a
proof, experts do not use the textbook rules. Instead they make leaps of inference to first
sketch out the key steps of the proof, and only then use the textbook rules to
fill in the details. I discovered that
such leaps of inference are enabled by informal, perceptually-based knowledge
representations and created a running cognitive model that plans proof
solutions like human experts. This
research 1) contributed to the understanding of how diagrammatic reasoning can aid reasoning and learning (Koedinger,
1992a), 2) provided a unified explanation of number of established characteristics
of expertise including perceptual
chunking and working forward bias
(Koedinger & Anderson, 1990), and 3) posed a challenge for learning
theories given the discovery of an alternative
perceptually based representation of expertise that does not clearly follow
from existing learning algorithms (Koedinger, 1992b).
This
basic research on expert thinking led to the design of a computer interface and
intelligent tutoring system to help students learn the implicit thinking steps
behind proof planning (Koedinger, 1991; Koedinger & Anderson, 1993a). This “plan reification” approach has worked
well in some cases, like the flow proof representation of search paths in the
ANGLE geometry tutor, but not in others, like the interface in an early
equation solving tutor (Anderson, Corbett, Koedinger & Pelletier,
1995). In the latter case, the resulting
pedagogical interface was not similar
enough to the target performance
interface, namely, solving equations on paper. Thus, it did not facilitate transfer of learning. The
design lesson is that a good pedagogical interface should help reveal the
thinking process, but not at the expense of obscuring the similarity to the
desired performance interface.
To extend and generalize the geometry work, I began research
in the algebra domain focusing on word problem solving. Like the process of constructing a proof, the
steps in translating problem situations to algebraic symbols are not explicitly
addressed in typical instruction. In one
experiment, we contrasted tutorial instruction using textbook problems with
“inductive support” instruction in which we placed more concrete “scaffolding”
questions in front of the more abstract goal of symbolizing an algebra story
problem.. W we found that students using
this inductive support version learned significantly more than students using
the textbook version (Koedinger & Anderson, 1998). This study and the geometry work provide
evidence for the inductive support hypothesis: Formal knowledge grows out of prior inductive experience which provides
a semantic foundation in the form of back-up strategies for retrieving,
checking, or rederiving formal knowledge.
Psychology PhD
students Kevin Gluck (1999) and Lisa Haverty pursued
this line of research. Haverty analyzed think-aloud protocols
and created a cognitive model of what students know and can learn about
inducing general rules from data (Haverty, Koedinger,
Klahr, & Alibali, 2000). Haverty’s thesis
followed up on the observation that numeric pattern knowledge was critical in
guiding the search for a general rule. Relevant to the debate about calculator use
in schools, she showed that improving students’ basic number fact knowledge
enhances their competence at the higher-order thinking task of inducing rules
from data.
Extensive experience
with building cognitive models through the time-consuming and somewhat
subjective process of think-aloud protocol analysis began to make me seek a
complementary methodology. I wanted a
methodology that: 1) could more quickly yield results that could inform
instructional design, 2) provide empirical data about the nature of knowledge
and performance that is less suspect to subjective protocol coding judgments,
and 3) not only reveal the thinking process, but also indicate something about
learning.
I developed a methodology called "Difficulty Factors Assessment" which we has now been used in numerous studies (Aleven, Koedinger, Sinclair, & Snyder, 1998; Baker, Corbett, Koedinger, & Schneider, 2003; Heffernan & Koedinger, 1997, 1998; Koedinger, Alibali, & Nathan, in progress; Koedinger, 2002; Koedinger & Cross, 2000; Koedinger & MacLaren, 1997; Koedinger & Nathan, 2004; Koedinger & Tabachneck, 1995; Tabachneck, Koedinger, & Nathan, 1995; Verzoni & Koedinger, 1997). In Difficulty Factors Assessment (DFA), we use theory and task analysis to generate hypotheses about the likely factors that cause student difficulties and then use these factors to systematically generate a pool of problems. The first DFA studies addressed the domain of early algebra problem solving. Based on our discovery of the important role of informal strategies in geometry proof and algebra symbolization, we hypothesized that students may be more likely to use invented strategies when problems are presented in a familiar context, like a word problem (e.g., “Ted works 6 hours and gets $66 in tips. If he made 81.90, what’s his hourly wage?”), than when presented in an analogous equation (e.g., x * 6 + 66 = 81.90). In fact, contrary to the predictions of mathematics teachers and educators (Nathan & Koedinger, 2000), we demonstrated that high school students are better at these simple algebra word problems than the corresponding equations (Koedinger & Nathan, 2004). We have replicated this result with a variety of student populations (Koedinger & MacLaren, 1997; Koedinger, Alibali, & Nathan, in progress; Koedinger & Tabachneck, 1995; Verzoni & Koedinger, 1997).
The
contrast between educator predictions and the students’ actual difficulties is
an instance of a more general phenomenon I call “expert blind spot”. As we develop expertise, for instance with
algebraic symbols, we lose the ability to accurately introspect on problem
difficulty from a student’s perspective.
Consistent with this notion, we found that teachers with the most
experience (high school math teachers) are the most likely to make the
incorrect prediction (Nathan, Koedinger, & Tabachneck,
2000, Nathan & Koedinger, 2000).
The expert blind spot
phenomenon has important implications not only for educational technology and
how it is evaluated, but also for software evaluation more generally. Software
designers who are expert programmers or experts in the end-user domain are
susceptible to expert blind spot and are likely to create applications that are
intuitive and easy for them to use,
but unintuitive and hard for learners who are novices in the interface or the
content domain. Because of expert
blind spot, seat-of-the-pants interface design is not likely to work and
empirical studies of users are critical.
The applied problem driving my research has been improving education, particularly of mathematics. The key vehicle for applying my basic research results to improving mathematics education has been intelligent tutoring systems. Individual human tutors have been shown to increase student achievement by as much as two standard deviations over traditional instruction. In evaluations of our intelligent tutors we have found about a one standard deviation improvement over traditional instruction.
My first studies with technology in real classrooms made
the importance of curriculum integration efforts plainly clear, both in a
quantitative statistical way and in a qualitative personal way. This classroom field study involved the ANGLE
tutor for geometry proof in which we designed interface notations to make more
concrete to students the inductive planning strategies of good problem solvers.
At the time of this study,
During this same
period of time, I taught two geometry classes myself. I experienced first hand the challenges urban
teachers face, not only in technology integration, but more generally. It was an eye opening experience that was
more mentally and emotionally draining than anything I have experienced before
or since. Both this teaching experience and the experimental results made clear
that we needed to pay closer attention to teacher needs if our tutors were
going to be generally effective.
In response, the subsequent PAT (Pump Algebra Tutor) project
(now known as “Cognitive Tutor Algebra”) took a client-centered design approach whereby we provided the technology
and psychological research, but received content guidance from educators,
teachers, and national standards
(Koedinger, Anderson, Hadley, & Mark, 1997). We teamed up with Bill
Hadley, a teacher writing a real-world problem-oriented algebra curriculum and
text designed to build on students’ common sense. We developed PAT together
with the curriculum so that they were tightly integrated from the start. We
also began an intensive teacher-training program. In Koedinger, Anderson,
Hadley, & Mark (1997), we reported on the first major field study of PAT,
which involved three Pittsburgh public schools and over 500 students. The study demonstrated dramatic effects of
the combination of the curriculum and PAT relative to traditional algebra
instruction. Students in experimental classes outperformed control classes by 100%
on assessments of the targeted problem solving and multiple representation
objectives (a one standard deviation effect).
They were also 15% better on basic skills as measured by standardized
test items from the Iowa and math SAT.
We later replicated these results in city schools in Pittsburgh and
Milwaukee showing an improvement range of 50-100% on the new standards
assessments and 15-25% on the standardized test items (Koedinger, Corbett,
Ritter, & Shapiro, 2000).
PAT is now the
most widely used Intelligent Tutoring System in K12 schools if not generally. Through
university-based dissemination efforts, PAT spread to some 75 urban and
suburban high schools, middle schools and colleges across the US by 1998. In that year, we formed a spin-off company, Carnegie Learning Inc., which has
taken over the dissemination and further research and development of PAT. As of
summer, 2004, Carnegie Learning has the course in over 1700 schools or over 5%
of the market. In Koedinger (2000) and
Corbett, Koedinger, & Hadley (2000) we have reviewed and reflected on the
factors that led to the widespread adoption and sustained use of the course.
Based
on the success of PAT, John Anderson, Albert Corbett, and I formed the
Pittsburgh Advanced Cognitive Tutor (PACT) Center, which was funded by
foundations local to Pittsburgh. The
goal was to create curriculum materials, training, and Cognitive Tutors for
three years of high school math and demonstrate a dramatic increase in high
school math achievement.
In the PACT project,
I directed the research and development of the Geometry Cognitive Tutor
course. We evaluated the impact of the
PACT project by comparing students taking the three-year sequence of Cognitive
Tutor courses with those in traditional math course sequence. We compared these two groups (Cognitive Tutor
vs. Traditional Math) at the end of both the 1997-98 and 1998-99 school
years. Our assessments included items
from the Third International Math and Science Study (TIMSS) and the math SAT.
In addition, students were asked to use graphs, tables and equations to model
real-world problem situations and answer relevant questions. The
results indicate that Cognitive Tutor students performed 30% better than
comparison students on the TIMSS questions and performed an average of 225%
better on the real-world problem solving assessment (Koedinger, Corbett,
Ritter, & Shapiro, 2000).
In the fall of 1999,
we began a new PACT Center project to develop Cognitive Tutor courses with the
three years of middle school math, 6th-8th grade. This three-year project was funded by a $2.7
million grant from our spin-off company Carnegie Learning. The resulting courses were shown to increase
student learning relative to existing courses (Koedinger, 2002) and were
licensed to Carnegie Learning.
Since
receiving tenure, I have engaged in two key efforts to expand the cognitive
modeling and Cognitive Tutor research agendas. First is an effort to extend
cognitive modeling and tutoring to explore students’ reflective learning or
meta-cognitive processes. Second is an
effort to facilitate other learning science researchers to use learning
technologies do the kind of advancement of learning theory and educational
applications for which we at CMU have been so successful.
My
collaborators and I have been exploring meta-cognition and tutoring in three
areas: self-explanation, error detection and correction, and help seeking.
Our
first results in addressing in meta-cognition in tutors are that we can enhance
cognitive tutors by having students provide “self-explanations” (rather than
just receiving teacher explanations) of the principles that justify
problem-solving steps. In particular,
the early work showed that a menu-based
“explanation by reference” approach, which is easily implemented in a
computer-based form, yields enhanced student learning and transfer (Aleven
& Koedinger, 2002). These results
are comparable to effects achieved in face-to-face instruction where the
self-explanations are in natural language.
Subsequent research has been advancing language technology to implement
computer-based understanding of student explanations provided in natural
language, with good success (Aleven, Popescu, & Koedinger, 2001, 2002;
Popescu & Koedinger, 2000). We have also become exploring whether this
addition of natural language explanation improves learning beyond the prior
explanation-by-reference approach (Aleven, Popescu, & Koedinger, 2003;
Aleven, Ogan, Torrey, & Koedinger, 2004).
Importantly, this line of research has significant overlap with the
recent joint HCII and LTI junior faculty hire Carolyn Rose.
A second target of research on
meta-cognition focuses on error detection and self-correction skills. The goal is to help students become more
flexible and adaptive "intelligent novices" by supporting learning of
interpretable declarative knowledge and more general interpretive procedures.
My HCI PhD student Santosh Mathan
performed experiments within a Cognitive Tutor for Microsoft Excel programming
comparing an “intelligent novice” tutor that provided feedback toward enhancing
error detection and self-correction skills with an “expert” tutor that provided
feedback toward not making errors in the first place. He found that the intelligent novice tutor led to better learning including better
transfer to novel problems and better longer-term retention of the
programming skills acquired (Mathan & Koedinger,
2003; Mathan, 2003).
He won a best student paper award
at the 2003 Artificial Intelligence in Education conference.
A
third target of research on meta-cognition focuses on student help-seeking
skills. This research was motivated by
an analysis of student-tutor interaction log files where we have identified
significant weaknesses in student’s meta-cognitive strategies for help
seeking. Contrary to the folk wisdom in the Intelligent Tutoring System field to
“maximize student control”, we found that many students lack the meta-cognitive
skills to seek help when they need it.
The paper on this work (Aleven & Koedinger, 2000b) won the best paper award at the Intelligent
Tutoring System 2000 Conference. We are
now working on a Meta-Cognitive Tutor to help students become better
help-seekers (Aleven, McLaren, Roll, & Koedinger, 2004).
Building
off past technical work on a plug-in tutor agent architecture (Ritter & Koedinger,
1997; Koedinger, Suthers, & Forbus,
1999), we began a project to create software tools to address the problem that
cognitive modeling and tutor development has been mostly limited to researchers
at CMU and, even here, it is time-consuming and requires unique expertise. The plug-in architecture made it possible to
combine our tutors in a component-based fashion with the best of existing
performance interfaces, like Microsoft Excel, and pedagogical interfaces, like
simulations and representation construction tools created by others. It also has facilitated us in creating a
suite of tools that build on existing off the shelf software to make cognitive
task analysis, cognitive modeling, and Cognitive Tutor development more
accessible. Our preliminary evidence is that the Cognitive Tutor Authoring Tools
may reduce development time by more than a factor of two (Koedinger,
Aleven, & Heffernan, 2003; Koedinger, Aleven, Heffernan, McLaren, & Hockenberry, 2004).
We have also demonstrated that non-programmers can create Cognitive
Tutor behavior in the form of Pseudo Tutors that behave just like Cognitive
Tutors but do not require the programming of a production system cognitive
model.
The
Cognitive Tutor Authoring Tools research agenda has been a central component in
a number of ongoing research efforts, including the CMU Open Learning
Initiative and the Assistment project, and a
particularly important new effort, the Pittsburgh Science of Learning Center.
The
CMU Open Learning Initiative is a project, funded by the Hewlett Foundation, to
use learning science and technology to create on-line courses at CMU that can
be widely distributed and help others get the benefits of a CMU education and
associated improvements, thereof, based on our unique scientific approach to
course evaluation, design, and development. The Cognitive Tutor Authoring Tools
have been used for rapid development of Pseudo Tutors, particularly for the
Economics and Statistics courses, and a full Cognitive Tutor for the Logic
course. Further work is planned to
support other courses, particularly tutoring of more interactive
problem-solving activities inside virtual laboratories in the Chemistry and
Causal Reasoning course.
We are using
the Cognitive Tutor Authoring Tools to create a web-based "Assistment" system for middle school mathematics. This
system will 1) quickly predict a student’s score on a standard-based test, 2)
provide feedback to teachers about how they can specifically adapt their
instruction to address student knowledge gaps, and 3) unlike other assessments
system, provide an opportunity for students to get intelligent tutoring
assistance at the same time as reliable assessment data is being collected
behind the scenes. In other words, Assistments provide instructional assistance as they perform assessment. The development of Assistments is motivated by the movement towards high
stakes testing (e.g., the No Child Left Behind act) that is being used in more
states as part of graduation requirements.
We are in the first year of a four year Department of Education grant to
perform a series of the experiments to investigate whether Assistments
can effectively predict state test performance and whether classroom use of the
student will lead to higher student achievement.
The
Pittsburgh Science of Learning Center (PSLC) is a new center, in partnership
with the University of Pittsburgh and Carnegie Learning, that I will
direct. The PSLC is funded by the National Science Foundation for 5 years and
about $25 million. It will provide a
novel research facility that will dramatically increase the ease and speed with
which learning researchers can create the rigorous, theory-based experiments
that will pave the way to an understanding of robust learning. This research
facility, called LearnLab, will be available
internationally, like a particle accelerator or a supercomputer center. The facility builds on the combined expertise
in Pittsburgh, on the human side in Cognitive Psychology, Developmental
Psychology, and Human-Computer Interaction, and on the technical side in
Intelligent Tutoring Systems, Machine Learning, and Language Technologies. PSLC
will make use of advanced technologies to facilitate the design of experiments
that combine the realism of classroom field studies and the rigor of controlled
theory-based laboratory studies. Use of technology-enhanced courses will produce
volumes of high-density data both on the short-term consequences of variation
in learning processes and on effects of these changes on “robust learning”,
that is, long-term retention, transfer, and accelerated future learning. PSLC
will create and support seven such full courses, two in high school
mathematics, two in college science, and three in college language learning.
PSLC researchers will
use LearnLab to create experiments embedded in
technology-enhanced courses that test learning principles designed to
facilitate robust learning. We will address several of the widely accepted, but
insufficiently studied “principles” enumerated in the well known National
Academy of Science’s book “How People Learn” such as “build on prior
knowledge”, “integrate conceptual and procedural knowledge”, and “encourage
meta-cognition”. Our analysis tools and learning-theory based software will
facilitate the creation of detailed computational theories for when, why, and
how these principles operate.
PSLC’s LearnLab will enrich the
scientific infrastructure by establishing unprecedented collaborations between
existing and future laboratory scientists, computer scientists and
instructional designers. The Center will
provide 50-100 pre-doctoral and post-doctoral students research experience in
developing and using LearnLab tools to address
challenging learning problems. PSLC will also have broad positive impacts on
schools. Experimental manipulations that
demonstrate substantial learning gains will be incorporated in LearnLab schools and eventually to many other schools. PSLC researchers in Pittsburgh and worldwide
will enhance scientific knowledge about robust human learning by 1) basing both
theory development and instructional design on close, detailed, study of
long-term student learning, 2) demonstrating the value of online intelligent
tutors as research platforms that support learning experiment and theory
development, 3) demonstrating a new synergy between use-driven research and
theory-testing instructional design.