Research
Statement for Kenneth R. Koedinger
Summer, 2004
My research
spans a number of areas within the fields of Cognitive Science, Cognitive
Psychology, Human-Computer Interaction, Artificial Intelligence, and Education.
Within Cognitive Psychology, my focus has been on the nature of complex human
problem solving and learning. I have
employed experimental methods and think aloud protocol studies to explore these
areas, but have especially used intelligent computer-based tutoring systems as
research vehicles to tightly control instructional manipulations, to run
learning studies of longer more realistic duration, and to collect detailed
micro-genetic data on the learning process.
Within Human-Computer Interaction and Education, I have coined the term
"expert blind spot" to summarize results of experiments showing that
domain experts can be systematically inaccurate in judging the difficulties of
novice users/students. Thus, designing
computer interfaces or instruction should not be based on intuition alone, but
should be informed by general psychological theory and domain specific
empirical data. I have developed methods
for efficiently collecting such data and for employing cognitive theory in the
design of Cognitive Tutors. I have run
laboratory and field experiments to demonstrate significant student learning
gains from Cognitive Tutors relative to alternative instructional methods. I have also helped to widely disseminate
these systems, so that, for instance, Cognitive Tutor Algebra is now in over 1700
schools across the
The first
sections of this statement are organized around my earlier pre-tenure research
goals:
1)
to make new
discoveries about the nature of complex problem solving and learning and
2)
to use these
discoveries in the design of intelligent tutoring systems and classroom
interventions that demonstrably increase student mathematics learning.
The final main
section focuses on my efforts, since tenure, to expand the cognitive modeling
and cognitive tutor research agendas. The
culmination of those efforts is the new Pittsburgh Science of Learning Center
that I will direct. The center is funded
by the National Science Foundation for 5 years and about $25 million and will
start in October, 2004.
My
basic research has investigated the role of informal, invented, or intuitive
knowledge in the learning of mathematical problem solving and reasoning. The well-structured and rule-bound surface of
mathematical reasoning gives the impression that the underlying thinking
process has the same structure and rules.
My basic research debunks this tempting impression and, furthermore,
illustrates what the rules of mathematical thinking are if they are not the rules of mathematics itself.
Research
on how people come to understand mathematical formalisms is not only directly
relevant to mathematics and science learning and instructional design, but it
is more generally relevant to the design of information technologies and their
human-computer interfaces.
In
my early research, I developed a cognitive theory of how good problem solvers
are able to make intuitive leaps of inference in discovering and creating proofs
of geometry theorems (Koedinger &
Anderson, 1990). A careful analysis of “think-aloud” reports of human experts
revealed that when planning a proof,
experts do not use the textbook rules.
Instead they make leaps of inference to first sketch out the key steps
of the proof, and only then use the textbook rules to fill in the details. I discovered that such leaps of inference are
enabled by informal, perceptually-based knowledge representations and created a
running cognitive model that plans proof solutions like human experts. This research 1) contributed to the
understanding of how diagrammatic
reasoning can aid reasoning and learning (Koedinger, 1992a), 2) provided a
unified explanation of number of established characteristics of expertise
including perceptual chunking and working forward bias (Koedinger &
Anderson, 1990), and 3) posed a challenge for learning theories given the
discovery of an alternative perceptually
based representation of expertise that does not clearly follow from existing
learning algorithms (Koedinger, 1992b).
This
basic research on expert thinking led to the design of a computer interface and
intelligent tutoring system to help students learn the implicit thinking steps
behind proof planning (Koedinger, 1991; Koedinger & Anderson, 1993a). This “plan reification” approach has worked
well in some cases, like the flow proof representation of search paths in the
ANGLE geometry tutor, but not in others, like the interface in an early
equation solving tutor (Anderson, Corbett, Koedinger & Pelletier,
1995). In the latter case, the resulting
pedagogical interface was not similar
enough to the target performance
interface, namely, solving equations on paper. Thus, it did not facilitate transfer of
learning. The design lesson is that a good pedagogical interface should help
reveal the thinking process, but not at the expense of obscuring the similarity
to the desired performance interface.
To extend and generalize the geometry work, I began research
in the algebra domain focusing on word problem solving. Like the process of constructing a proof, the
steps in translating problem situations to algebraic symbols are not explicitly
addressed in typical instruction. In one
experiment, we contrasted tutorial instruction using textbook problems with
“inductive support” instruction in which we placed more concrete “scaffolding”
questions in front of the more abstract goal of symbolizing an algebra story
problem.. W we found that students using
this inductive support version learned significantly more than students using
the textbook version (Koedinger & Anderson, 1998). This study and the geometry work provide
evidence for the inductive support hypothesis: Formal knowledge grows out of prior inductive experience which provides
a semantic foundation in the form of back-up strategies for retrieving,
checking, or rederiving formal knowledge.
Psychology PhD
students Kevin Gluck (1999) and Lisa Haverty pursued this line of research.
Haverty analyzed think-aloud protocols and created a cognitive model of what
students know and can learn about inducing general rules from data (Haverty,
Koedinger, Klahr, & Alibali, 2000).
Haverty’s thesis followed up on the observation that numeric pattern
knowledge was critical in guiding the search for a general rule. Relevant
to the debate about calculator use in schools, she showed that improving
students’ basic number fact knowledge enhances their competence at the
higher-order thinking task of inducing rules from data.
Extensive experience
with building cognitive models through the time-consuming and somewhat
subjective process of think-aloud protocol analysis began to make me seek a
complementary methodology. I wanted a
methodology that: 1) could more quickly yield results that could inform
instructional design, 2) provide empirical data about the nature of knowledge
and performance that is less suspect to subjective protocol coding judgments,
and 3) not only reveal the thinking process, but also indicate something about
learning.
I developed a methodology called "Difficulty Factors Assessment" which we has now been used in numerous studies (Aleven, Koedinger, Sinclair, & Snyder, 1998; Baker, Corbett, Koedinger, & Schneider, 2003; Heffernan & Koedinger, 1997, 1998; Koedinger, Alibali, & Nathan, in progress; Koedinger, 2002; Koedinger & Cross, 2000; Koedinger & MacLaren, 1997; Koedinger & Nathan, 2004; Koedinger & Tabachneck, 1995; Tabachneck, Koedinger, & Nathan, 1995; Verzoni & Koedinger, 1997). In Difficulty Factors Assessment (DFA), we use theory and task analysis to generate hypotheses about the likely factors that cause student difficulties and then use these factors to systematically generate a pool of problems. The first DFA studies addressed the domain of early algebra problem solving. Based on our discovery of the important role of informal strategies in geometry proof and algebra symbolization, we hypothesized that students may be more likely to use invented strategies when problems are presented in a familiar context, like a word problem (e.g., “Ted works 6 hours and gets $66 in tips. If he made 81.90, what’s his hourly wage?”), than when presented in an analogous equation (e.g., x * 6 + 66 = 81.90). In fact, contrary to the predictions of mathematics teachers and educators (Nathan & Koedinger, 2000), we demonstrated that high school students are better at these simple algebra word problems than the corresponding equations (Koedinger & Nathan, 2004). We have replicated this result with a variety of student populations (Koedinger & MacLaren, 1997; Koedinger, Alibali, & Nathan, in progress; Koedinger & Tabachneck, 1995; Verzoni & Koedinger, 1997).
The
contrast between educator predictions and the students’ actual difficulties is
an instance of a more general phenomenon I call “expert blind spot”. As we develop expertise, for instance with
algebraic symbols, we lose the ability to accurately introspect on problem
difficulty from a student’s perspective.
Consistent with this notion, we found that teachers with the most
experience (high school math teachers) are the most likely to make the
incorrect prediction (Nathan, Koedinger, & Tabachneck, 2000, Nathan &
Koedinger, 2000).
The expert blind spot
phenomenon has important implications not only for educational technology and
how it is evaluated, but also for software evaluation more generally. Software
designers who are expert programmers or experts in the end-user domain are
susceptible to expert blind spot and are likely to create applications that are
intuitive and easy for them to use,
but unintuitive and hard for learners who are novices in the interface or the
content domain. Because of expert
blind spot, seat-of-the-pants interface design is not likely to work and
empirical studies of users are critical.
The applied problem driving my research has been improving education, particularly of mathematics. The key vehicle for applying my basic research results to improving mathematics education has been intelligent tutoring systems. Individual human tutors have been shown to increase student achievement by as much as two standard deviations over traditional instruction. In evaluations of our intelligent tutors we have found about a one standard deviation improvement over traditional instruction.
My first studies with technology in real classrooms made
the importance of curriculum integration efforts plainly clear, both in a
quantitative statistical way and in a qualitative personal way. This classroom field study involved the ANGLE
tutor for geometry proof in which we designed interface notations to make more
concrete to students the inductive planning strategies of good problem solvers.
At the time of this study,
During this same
period of time, I taught two geometry classes myself. I experienced first hand the challenges urban
teachers face, not only in technology integration, but more generally. It was an eye opening experience that was
more mentally and emotionally draining than anything I have experienced before
or since. Both this teaching experience and the experimental results made clear
that we needed to pay closer attention to teacher needs if our tutors were
going to be generally effective.
In response, the subsequent PAT (Pump Algebra Tutor) project
(now known as “Cognitive Tutor Algebra”) took a client-centered design approach whereby we provided the technology
and psychological research, but received content guidance from educators,
teachers, and national standards
(Koedinger, Anderson, Hadley, & Mark, 1997). We teamed up with Bill
Hadley, a teacher writing a real-world problem-oriented algebra curriculum and
text designed to build on students’ common sense. We developed PAT together
with the curriculum so that they were tightly integrated from the start. We
also began an intensive teacher-training program. In Koedinger, Anderson,
Hadley, & Mark (1997), we reported on the first major field study of PAT,
which involved three Pittsburgh public schools and over 500 students. The study demonstrated dramatic effects of
the combination of the curriculum and PAT relative to traditional algebra
instruction. Students in experimental classes outperformed control classes by 100%
on assessments of the targeted problem solving and multiple representation
objectives (a one standard deviation effect). They were also 15% better on basic skills as
measured by standardized test items from the Iowa and math SAT. We later replicated these results in city
schools in Pittsburgh and Milwaukee showing an improvement range of 50-100% on
the new standards assessments and 15-25% on the standardized test items
(Koedinger, Corbett, Ritter, & Shapiro, 2000).
PAT is now the
most widely used Intelligent Tutoring System in K12 schools if not generally.
Through university-based dissemination efforts, PAT spread to some 75
urban and suburban high schools, middle schools and colleges across the US by
1998. In that year, we formed a spin-off company, Carnegie Learning Inc., which has
taken over the dissemination and further research and development of PAT. As of
summer, 2004, Carnegie Learning has the course in over 1700 schools or over 5%
of the market. In Koedinger (2000) and
Corbett, Koedinger, & Hadley (2000) we have reviewed and reflected on the
factors that led to the widespread adoption and sustained use of the course.
Based
on the success of PAT, John Anderson, Albert Corbett, and I formed the
Pittsburgh Advanced Cognitive Tutor (PACT) Center, which was funded by
foundations local to Pittsburgh. The
goal was to create curriculum materials, training, and Cognitive Tutors for
three years of high school math and demonstrate a dramatic increase in high
school math achievement.
In the PACT project,
I directed the research and development of the Geometry Cognitive Tutor
course. We evaluated the impact of the
PACT project by comparing students taking the three-year sequence of Cognitive
Tutor courses with those in traditional math course sequence. We compared these two groups (Cognitive Tutor
vs. Traditional Math) at the end of both the 1997-98 and 1998-99 school
years. Our assessments included items
from the Third International Math and Science Study (TIMSS) and the math SAT.
In addition, students were asked to use graphs, tables and equations to model
real-world problem situations and answer relevant questions. The
results indicate that Cognitive Tutor students performed 30% better than
comparison students on the TIMSS questions and performed an average of 225%
better on the real-world problem solving assessment (Koedinger, Corbett,
Ritter, & Shapiro, 2000).
In the fall of 1999,
we began a new PACT Center project to develop Cognitive Tutor courses with the
three years of middle school math, 6th-8th grade. This three-year project was funded by a $2.7
million grant from our spin-off company Carnegie Learning. The resulting courses were shown to increase
student learning relative to existing courses (Koedinger, 2002) and were
licensed to Carnegie Learning.
Since receiving
tenure, I have engaged in two key efforts to expand the cognitive modeling and
Cognitive Tutor research agendas. First is an effort to extend cognitive
modeling and tutoring to explore students’ reflective learning or
meta-cognitive processes. Second is an
effort to facilitate other learning science researchers to use learning
technologies do the kind of advancement of learning theory and educational
applications for which we at CMU have been so successful.
My
collaborators and I have been exploring meta-cognition and tutoring in three
areas: self-explanation, error detection and correction, and help seeking.
Our
first results in addressing in meta-cognition in tutors are that we can enhance
cognitive tutors by having students provide “self-explanations” (rather than
just receiving teacher explanations) of the principles that justify
problem-solving steps. In particular,
the early work showed that a menu-based
“explanation by reference” approach, which is easily implemented in a
computer-based form, yields enhanced student learning and transfer (Aleven
& Koedinger, 2002). These results
are comparable to effects achieved in face-to-face instruction where the
self-explanations are in natural language.
Subsequent research has been advancing language technology to implement
computer-based understanding of student explanations provided in natural
language, with good success (Aleven, Popescu, & Koedinger, 2001, 2002;
Popescu & Koedinger, 2000). We have also become exploring whether this
addition of natural language explanation improves learning beyond the prior
explanation-by-reference approach (Aleven, Popescu, & Koedinger, 2003;
Aleven, Ogan, Torrey, & Koedinger, 2004).
Importantly, this line of research has significant overlap with the
recent joint HCII and LTI junior faculty hire Carolyn Rose.
A second target of research on
meta-cognition focuses on error detection and self-correction skills. The goal is to help students become more
flexible and adaptive "intelligent novices" by supporting learning of
interpretable declarative knowledge and more general interpretive procedures.
My HCI PhD student Santosh Mathan performed experiments within a Cognitive
Tutor for Microsoft Excel programming comparing an “intelligent novice” tutor
that provided feedback toward enhancing error detection and self-correction
skills with an “expert” tutor that provided feedback toward not making errors
in the first place. He found that the intelligent novice tutor led to better
learning including better transfer to novel problems and better longer-term
retention of the programming skills acquired (Mathan & Koedinger, 2003;
Mathan, 2003). He won a best student paper award at the 2003
Artificial Intelligence in Education conference.
A
third target of research on meta-cognition focuses on student help-seeking
skills. This research was motivated by
an analysis of student-tutor interaction log files where we have identified
significant weaknesses in student’s meta-cognitive strategies for help
seeking. Contrary to the folk wisdom in the Intelligent Tutoring System field to
“maximize student control”, we found that many students lack the meta-cognitive
skills to seek help when they need it.
The paper on this work (Aleven & Koedinger, 2000b) won the best paper award at the Intelligent
Tutoring System 2000 Conference. We are
now working on a Meta-Cognitive Tutor to help students become better
help-seekers (Aleven, McLaren, Roll, & Koedinger, 2004).
Building
off past technical work on a plug-in tutor agent architecture (Ritter &
Koedinger, 1997; Koedinger, Suthers, & Forbus, 1999), we began a project to
create software tools to address the problem that cognitive modeling and tutor
development has been mostly limited to researchers at CMU and, even here, it is
time-consuming and requires unique expertise.
The plug-in architecture made it possible to combine our tutors in a
component-based fashion with the best of existing performance interfaces, like
Microsoft Excel, and pedagogical interfaces, like simulations and
representation construction tools created by others. It also has facilitated us in creating a suite
of tools that build on existing off the shelf software to make cognitive task
analysis, cognitive modeling, and Cognitive Tutor development more
accessible. Our preliminary evidence is that the Cognitive Tutor Authoring Tools
may reduce development time by more than a factor of two (Koedinger,
Aleven, & Heffernan, 2003; Koedinger, Aleven, Heffernan, McLaren, &
Hockenberry, 2004). We have also
demonstrated that non-programmers can create Cognitive Tutor behavior in the
form of Pseudo Tutors that behave just like Cognitive Tutors but do not require
the programming of a production system cognitive model.
The
Cognitive Tutor Authoring Tools research agenda has been a central component in
a number of ongoing research efforts, including the CMU Open Learning Initiative
and the Assistment project, and a particularly important new effort, the
Pittsburgh Science of Learning Center.
The
CMU Open Learning Initiative is a project, funded by the Hewlett Foundation, to
use learning science and technology to create on-line courses at CMU that can
be widely distributed and help others get the benefits of a CMU education and
associated improvements, thereof, based on our unique scientific approach to
course evaluation, design, and development. The Cognitive Tutor Authoring Tools
have been used for rapid development of Pseudo Tutors, particularly for the
Economics and Statistics courses, and a full Cognitive Tutor for the Logic
course. Further work is planned to support
other courses, particularly tutoring of more interactive problem-solving
activities inside virtual laboratories in the Chemistry and Causal Reasoning
course.
We are using
the Cognitive Tutor Authoring Tools to create a web-based
"Assistment" system for middle school mathematics. This system will
1) quickly predict a student’s score on a standard-based test, 2) provide
feedback to teachers about how they can specifically adapt their instruction to
address student knowledge gaps, and 3) unlike other assessments system, provide
an opportunity for students to get intelligent tutoring assistance at the same
time as reliable assessment data is being collected behind the scenes. In other words, Assistments provide
instructional assistance as they
perform assessment. The development of Assistments is motivated
by the movement towards high stakes testing (e.g., the No Child Left Behind
act) that is being used in more states as part of graduation requirements. We are in the first year of a four year
Department of Education grant to perform a series of the experiments to
investigate whether Assistments can effectively predict state test performance
and whether classroom use of the student will lead to higher student
achievement.
The
Pittsburgh Science of Learning Center (PSLC) is a new center, in partnership
with the University of Pittsburgh and Carnegie Learning, that I will
direct. The PSLC is funded by the National Science Foundation for 5 years and
about $25 million. It will provide a
novel research facility that will dramatically increase the ease and speed with
which learning researchers can create the rigorous, theory-based experiments
that will pave the way to an understanding of robust learning. This research
facility, called LearnLab, will be available internationally, like a particle
accelerator or a supercomputer center.
The facility builds on the combined expertise in Pittsburgh, on the
human side in Cognitive Psychology, Developmental Psychology, and
Human-Computer Interaction, and on the technical side in Intelligent Tutoring
Systems, Machine Learning, and Language Technologies. PSLC will make use of
advanced technologies to facilitate the design of experiments that combine the
realism of classroom field studies and the rigor of controlled theory-based
laboratory studies. Use of technology-enhanced courses will produce volumes of
high-density data both on the short-term consequences of variation in learning
processes and on effects of these changes on “robust learning”, that is,
long-term retention, transfer, and accelerated future learning. PSLC will
create and support seven such full courses, two in high school mathematics, two
in college science, and three in college language learning.
PSLC researchers will
use LearnLab to create experiments embedded in technology-enhanced courses that
test learning principles designed to facilitate robust learning. We will
address several of the widely accepted, but insufficiently studied “principles”
enumerated in the well known National Academy of Science’s book “How People
Learn” such as “build on prior knowledge”, “integrate conceptual and procedural
knowledge”, and “encourage meta-cognition”. Our analysis tools and
learning-theory based software will facilitate the creation of detailed
computational theories for when, why, and how these principles operate.
PSLC’s LearnLab will
enrich the scientific infrastructure by establishing unprecedented
collaborations between existing and future laboratory scientists, computer
scientists and instructional designers.
The Center will provide 50-100 pre-doctoral and post-doctoral students
research experience in developing and using LearnLab tools to address
challenging learning problems. PSLC will also have broad positive impacts on
schools. Experimental manipulations that
demonstrate substantial learning gains will be incorporated in LearnLab schools
and eventually to many other schools.
PSLC researchers in Pittsburgh and worldwide will enhance scientific
knowledge about robust human learning by 1) basing both theory development and
instructional design on close, detailed, study of long-term student learning,
2) demonstrating the value of online intelligent tutors as research platforms
that support learning experiment and theory development, 3) demonstrating a new
synergy between use-driven research and theory-testing instructional design.