Director of the Pittsburgh Science of Learning Center
spans a number of areas within the fields of Cognitive Science, Cognitive
Psychology, Human-Computer Interaction, Artificial Intelligence, and Education.
Within Cognitive Psychology, my focus has been on the nature of complex human
problem solving and learning. I have
employed experimental methods and think aloud protocol studies to explore these
areas, but have especially used intelligent computer-based tutoring systems as
research vehicles to tightly control instructional manipulations, to run
learning studies of longer more realistic duration, and to collect detailed
micro-genetic data on the learning process.
Within Human-Computer Interaction and Education, I have coined the term
"expert blind spot" to summarize results of experiments showing that
domain experts can be systematically inaccurate in judging the difficulties of
novice users/students. Thus, designing
computer interfaces or instruction should not be based on intuition alone, but
should be informed by general psychological theory and domain specific
empirical data. I have developed methods
for efficiently collecting such data and for employing cognitive theory in the
design of Cognitive Tutors. I have run
laboratory and field experiments to demonstrate significant student learning
gains from Cognitive Tutors relative to alternative instructional methods. I have also helped to widely disseminate
these systems, so that, for instance, Cognitive Tutor Algebra is now in over
1700 schools across the
The first sections of this statement are organized around my earlier pre-tenure research goals:
1) to make new discoveries about the nature of complex problem solving and learning and
2) to use these discoveries in the design of intelligent tutoring systems and classroom interventions that demonstrably increase student mathematics learning.
The final main section focuses on my efforts, since tenure, to expand the cognitive modeling and cognitive tutor research agendas. The culmination of those efforts is the new Pittsburgh Science of Learning Center that I will direct. The center is funded by the National Science Foundation for 5 years and about $25 million and will start in October, 2004.
My basic research has investigated the role of informal, invented, or intuitive knowledge in the learning of mathematical problem solving and reasoning. The well-structured and rule-bound surface of mathematical reasoning gives the impression that the underlying thinking process has the same structure and rules. My basic research debunks this tempting impression and, furthermore, illustrates what the rules of mathematical thinking are if they are not the rules of mathematics itself.
Research on how people come to understand mathematical formalisms is not only directly relevant to mathematics and science learning and instructional design, but it is more generally relevant to the design of information technologies and their human-computer interfaces.
In my early research, I developed a cognitive theory of how good problem solvers are able to make intuitive leaps of inference in discovering and creating proofs of geometry theorems (Koedinger & Anderson, 1990). A careful analysis of “think-aloud” reports of human experts revealed that when planning a proof, experts do not use the textbook rules. Instead they make leaps of inference to first sketch out the key steps of the proof, and only then use the textbook rules to fill in the details. I discovered that such leaps of inference are enabled by informal, perceptually-based knowledge representations and created a running cognitive model that plans proof solutions like human experts. This research 1) contributed to the understanding of how diagrammatic reasoning can aid reasoning and learning (Koedinger, 1992a), 2) provided a unified explanation of number of established characteristics of expertise including perceptual chunking and working forward bias (Koedinger & Anderson, 1990), and 3) posed a challenge for learning theories given the discovery of an alternative perceptually based representation of expertise that does not clearly follow from existing learning algorithms (Koedinger, 1992b).
This basic research on expert thinking led to the design of a computer interface and intelligent tutoring system to help students learn the implicit thinking steps behind proof planning (Koedinger, 1991; Koedinger & Anderson, 1993a). This “plan reification” approach has worked well in some cases, like the flow proof representation of search paths in the ANGLE geometry tutor, but not in others, like the interface in an early equation solving tutor (Anderson, Corbett, Koedinger & Pelletier, 1995). In the latter case, the resulting pedagogical interface was not similar enough to the target performance interface, namely, solving equations on paper. Thus, it did not facilitate transfer of learning. The design lesson is that a good pedagogical interface should help reveal the thinking process, but not at the expense of obscuring the similarity to the desired performance interface.
To extend and generalize the geometry work, I began research in the algebra domain focusing on word problem solving. Like the process of constructing a proof, the steps in translating problem situations to algebraic symbols are not explicitly addressed in typical instruction. In one experiment, we contrasted tutorial instruction using textbook problems with “inductive support” instruction in which we placed more concrete “scaffolding” questions in front of the more abstract goal of symbolizing an algebra story problem.. W we found that students using this inductive support version learned significantly more than students using the textbook version (Koedinger & Anderson, 1998). This study and the geometry work provide evidence for the inductive support hypothesis: Formal knowledge grows out of prior inductive experience which provides a semantic foundation in the form of back-up strategies for retrieving, checking, or rederiving formal knowledge.
Psychology PhD students Kevin Gluck (1999) and Lisa Haverty pursued this line of research. Haverty analyzed think-aloud protocols and created a cognitive model of what students know and can learn about inducing general rules from data (Haverty, Koedinger, Klahr, & Alibali, 2000). Haverty’s thesis followed up on the observation that numeric pattern knowledge was critical in guiding the search for a general rule. Relevant to the debate about calculator use in schools, she showed that improving students’ basic number fact knowledge enhances their competence at the higher-order thinking task of inducing rules from data.
Extensive experience with building cognitive models through the time-consuming and somewhat subjective process of think-aloud protocol analysis began to make me seek a complementary methodology. I wanted a methodology that: 1) could more quickly yield results that could inform instructional design, 2) provide empirical data about the nature of knowledge and performance that is less suspect to subjective protocol coding judgments, and 3) not only reveal the thinking process, but also indicate something about learning.
I developed a methodology called "Difficulty Factors Assessment" which we has now been used in numerous studies (Aleven, Koedinger, Sinclair, & Snyder, 1998; Baker, Corbett, Koedinger, & Schneider, 2003; Heffernan & Koedinger, 1997, 1998; Koedinger, Alibali, & Nathan, in progress; Koedinger, 2002; Koedinger & Cross, 2000; Koedinger & MacLaren, 1997; Koedinger & Nathan, 2004; Koedinger & Tabachneck, 1995; Tabachneck, Koedinger, & Nathan, 1995; Verzoni & Koedinger, 1997). In Difficulty Factors Assessment (DFA), we use theory and task analysis to generate hypotheses about the likely factors that cause student difficulties and then use these factors to systematically generate a pool of problems. The first DFA studies addressed the domain of early algebra problem solving. Based on our discovery of the important role of informal strategies in geometry proof and algebra symbolization, we hypothesized that students may be more likely to use invented strategies when problems are presented in a familiar context, like a word problem (e.g., “Ted works 6 hours and gets $66 in tips. If he made 81.90, what’s his hourly wage?”), than when presented in an analogous equation (e.g., x * 6 + 66 = 81.90). In fact, contrary to the predictions of mathematics teachers and educators (Nathan & Koedinger, 2000), we demonstrated that high school students are better at these simple algebra word problems than the corresponding equations (Koedinger & Nathan, 2004). We have replicated this result with a variety of student populations (Koedinger & MacLaren, 1997; Koedinger, Alibali, & Nathan, in progress; Koedinger & Tabachneck, 1995; Verzoni & Koedinger, 1997).
The contrast between educator predictions and the students’ actual difficulties is an instance of a more general phenomenon I call “expert blind spot”. As we develop expertise, for instance with algebraic symbols, we lose the ability to accurately introspect on problem difficulty from a student’s perspective. Consistent with this notion, we found that teachers with the most experience (high school math teachers) are the most likely to make the incorrect prediction (Nathan, Koedinger, & Tabachneck, 2000, Nathan & Koedinger, 2000).
The expert blind spot phenomenon has important implications not only for educational technology and how it is evaluated, but also for software evaluation more generally. Software designers who are expert programmers or experts in the end-user domain are susceptible to expert blind spot and are likely to create applications that are intuitive and easy for them to use, but unintuitive and hard for learners who are novices in the interface or the content domain. Because of expert blind spot, seat-of-the-pants interface design is not likely to work and empirical studies of users are critical.
The applied problem driving my research has been improving education, particularly of mathematics. The key vehicle for applying my basic research results to improving mathematics education has been intelligent tutoring systems. Individual human tutors have been shown to increase student achievement by as much as two standard deviations over traditional instruction. In evaluations of our intelligent tutors we have found about a one standard deviation improvement over traditional instruction.
My first studies with technology in real classrooms made
the importance of curriculum integration efforts plainly clear, both in a
quantitative statistical way and in a qualitative personal way. This classroom field study involved the ANGLE
tutor for geometry proof in which we designed interface notations to make more
concrete to students the inductive planning strategies of good problem solvers.
At the time of this study,
During this same period of time, I taught two geometry classes myself. I experienced first hand the challenges urban teachers face, not only in technology integration, but more generally. It was an eye opening experience that was more mentally and emotionally draining than anything I have experienced before or since. Both this teaching experience and the experimental results made clear that we needed to pay closer attention to teacher needs if our tutors were going to be generally effective.
In response, the subsequent PAT (Pump Algebra Tutor) project (now known as “Cognitive Tutor Algebra”) took a client-centered design approach whereby we provided the technology and psychological research, but received content guidance from educators, teachers, and national standards (Koedinger, Anderson, Hadley, & Mark, 1997). We teamed up with Bill Hadley, a teacher writing a real-world problem-oriented algebra curriculum and text designed to build on students’ common sense. We developed PAT together with the curriculum so that they were tightly integrated from the start. We also began an intensive teacher-training program. In Koedinger, Anderson, Hadley, & Mark (1997), we reported on the first major field study of PAT, which involved three Pittsburgh public schools and over 500 students. The study demonstrated dramatic effects of the combination of the curriculum and PAT relative to traditional algebra instruction. Students in experimental classes outperformed control classes by 100% on assessments of the targeted problem solving and multiple representation objectives (a one standard deviation effect). They were also 15% better on basic skills as measured by standardized test items from the Iowa and math SAT. We later replicated these results in city schools in Pittsburgh and Milwaukee showing an improvement range of 50-100% on the new standards assessments and 15-25% on the standardized test items (Koedinger, Corbett, Ritter, & Shapiro, 2000).
PAT is now the most widely used Intelligent Tutoring System in K12 schools if not generally. Through university-based dissemination efforts, PAT spread to some 75 urban and suburban high schools, middle schools and colleges across the US by 1998. In that year, we formed a spin-off company, Carnegie Learning Inc., which has taken over the dissemination and further research and development of PAT. As of summer, 2004, Carnegie Learning has the course in over 1700 schools or over 5% of the market. In Koedinger (2000) and Corbett, Koedinger, & Hadley (2000) we have reviewed and reflected on the factors that led to the widespread adoption and sustained use of the course.
Based on the success of PAT, John Anderson, Albert Corbett, and I formed the Pittsburgh Advanced Cognitive Tutor (PACT) Center, which was funded by foundations local to Pittsburgh. The goal was to create curriculum materials, training, and Cognitive Tutors for three years of high school math and demonstrate a dramatic increase in high school math achievement.
In the PACT project, I directed the research and development of the Geometry Cognitive Tutor course. We evaluated the impact of the PACT project by comparing students taking the three-year sequence of Cognitive Tutor courses with those in traditional math course sequence. We compared these two groups (Cognitive Tutor vs. Traditional Math) at the end of both the 1997-98 and 1998-99 school years. Our assessments included items from the Third International Math and Science Study (TIMSS) and the math SAT. In addition, students were asked to use graphs, tables and equations to model real-world problem situations and answer relevant questions. The results indicate that Cognitive Tutor students performed 30% better than comparison students on the TIMSS questions and performed an average of 225% better on the real-world problem solving assessment (Koedinger, Corbett, Ritter, & Shapiro, 2000).
In the fall of 1999, we began a new PACT Center project to develop Cognitive Tutor courses with the three years of middle school math, 6th-8th grade. This three-year project was funded by a $2.7 million grant from our spin-off company Carnegie Learning. The resulting courses were shown to increase student learning relative to existing courses (Koedinger, 2002) and were licensed to Carnegie Learning.
Since receiving tenure, I have engaged in two key efforts to expand the cognitive modeling and Cognitive Tutor research agendas. First is an effort to extend cognitive modeling and tutoring to explore students’ reflective learning or meta-cognitive processes. Second is an effort to facilitate other learning science researchers to use learning technologies do the kind of advancement of learning theory and educational applications for which we at CMU have been so successful.
My collaborators and I have been exploring meta-cognition and tutoring in three areas: self-explanation, error detection and correction, and help seeking.
Our first results in addressing in meta-cognition in tutors are that we can enhance cognitive tutors by having students provide “self-explanations” (rather than just receiving teacher explanations) of the principles that justify problem-solving steps. In particular, the early work showed that a menu-based “explanation by reference” approach, which is easily implemented in a computer-based form, yields enhanced student learning and transfer (Aleven & Koedinger, 2002). These results are comparable to effects achieved in face-to-face instruction where the self-explanations are in natural language. Subsequent research has been advancing language technology to implement computer-based understanding of student explanations provided in natural language, with good success (Aleven, Popescu, & Koedinger, 2001, 2002; Popescu & Koedinger, 2000). We have also become exploring whether this addition of natural language explanation improves learning beyond the prior explanation-by-reference approach (Aleven, Popescu, & Koedinger, 2003; Aleven, Ogan, Torrey, & Koedinger, 2004). Importantly, this line of research has significant overlap with the recent joint HCII and LTI junior faculty hire Carolyn Rose.
A second target of research on meta-cognition focuses on error detection and self-correction skills. The goal is to help students become more flexible and adaptive "intelligent novices" by supporting learning of interpretable declarative knowledge and more general interpretive procedures. My HCI PhD student Santosh Mathan performed experiments within a Cognitive Tutor for Microsoft Excel programming comparing an “intelligent novice” tutor that provided feedback toward enhancing error detection and self-correction skills with an “expert” tutor that provided feedback toward not making errors in the first place. He found that the intelligent novice tutor led to better learning including better transfer to novel problems and better longer-term retention of the programming skills acquired (Mathan & Koedinger, 2003; Mathan, 2003). He won a best student paper award at the 2003 Artificial Intelligence in Education conference.
A third target of research on meta-cognition focuses on student help-seeking skills. This research was motivated by an analysis of student-tutor interaction log files where we have identified significant weaknesses in student’s meta-cognitive strategies for help seeking. Contrary to the folk wisdom in the Intelligent Tutoring System field to “maximize student control”, we found that many students lack the meta-cognitive skills to seek help when they need it. The paper on this work (Aleven & Koedinger, 2000b) won the best paper award at the Intelligent Tutoring System 2000 Conference. We are now working on a Meta-Cognitive Tutor to help students become better help-seekers (Aleven, McLaren, Roll, & Koedinger, 2004).
Building off past technical work on a plug-in tutor agent architecture (Ritter & Koedinger, 1997; Koedinger, Suthers, & Forbus, 1999), we began a project to create software tools to address the problem that cognitive modeling and tutor development has been mostly limited to researchers at CMU and, even here, it is time-consuming and requires unique expertise. The plug-in architecture made it possible to combine our tutors in a component-based fashion with the best of existing performance interfaces, like Microsoft Excel, and pedagogical interfaces, like simulations and representation construction tools created by others. It also has facilitated us in creating a suite of tools that build on existing off the shelf software to make cognitive task analysis, cognitive modeling, and Cognitive Tutor development more accessible. Our preliminary evidence is that the Cognitive Tutor Authoring Tools may reduce development time by more than a factor of two (Koedinger, Aleven, & Heffernan, 2003; Koedinger, Aleven, Heffernan, McLaren, & Hockenberry, 2004). We have also demonstrated that non-programmers can create Cognitive Tutor behavior in the form of Pseudo Tutors that behave just like Cognitive Tutors but do not require the programming of a production system cognitive model.
The Cognitive Tutor Authoring Tools research agenda has been a central component in a number of ongoing research efforts, including the CMU Open Learning Initiative and the Assistment project, and a particularly important new effort, the Pittsburgh Science of Learning Center.
The CMU Open Learning Initiative is a project, funded by the Hewlett Foundation, to use learning science and technology to create on-line courses at CMU that can be widely distributed and help others get the benefits of a CMU education and associated improvements, thereof, based on our unique scientific approach to course evaluation, design, and development. The Cognitive Tutor Authoring Tools have been used for rapid development of Pseudo Tutors, particularly for the Economics and Statistics courses, and a full Cognitive Tutor for the Logic course. Further work is planned to support other courses, particularly tutoring of more interactive problem-solving activities inside virtual laboratories in the Chemistry and Causal Reasoning course.
We are using the Cognitive Tutor Authoring Tools to create a web-based "Assistment" system for middle school mathematics. This system will 1) quickly predict a student’s score on a standard-based test, 2) provide feedback to teachers about how they can specifically adapt their instruction to address student knowledge gaps, and 3) unlike other assessments system, provide an opportunity for students to get intelligent tutoring assistance at the same time as reliable assessment data is being collected behind the scenes. In other words, Assistments provide instructional assistance as they perform assessment. The development of Assistments is motivated by the movement towards high stakes testing (e.g., the No Child Left Behind act) that is being used in more states as part of graduation requirements. We are in the first year of a four year Department of Education grant to perform a series of the experiments to investigate whether Assistments can effectively predict state test performance and whether classroom use of the student will lead to higher student achievement.
The Pittsburgh Science of Learning Center (PSLC) is a new center, in partnership with the University of Pittsburgh and Carnegie Learning, that I will direct. The PSLC is funded by the National Science Foundation for 5 years and about $25 million. It will provide a novel research facility that will dramatically increase the ease and speed with which learning researchers can create the rigorous, theory-based experiments that will pave the way to an understanding of robust learning. This research facility, called LearnLab, will be available internationally, like a particle accelerator or a supercomputer center. The facility builds on the combined expertise in Pittsburgh, on the human side in Cognitive Psychology, Developmental Psychology, and Human-Computer Interaction, and on the technical side in Intelligent Tutoring Systems, Machine Learning, and Language Technologies. PSLC will make use of advanced technologies to facilitate the design of experiments that combine the realism of classroom field studies and the rigor of controlled theory-based laboratory studies. Use of technology-enhanced courses will produce volumes of high-density data both on the short-term consequences of variation in learning processes and on effects of these changes on “robust learning”, that is, long-term retention, transfer, and accelerated future learning. PSLC will create and support seven such full courses, two in high school mathematics, two in college science, and three in college language learning.
PSLC researchers will use LearnLab to create experiments embedded in technology-enhanced courses that test learning principles designed to facilitate robust learning. We will address several of the widely accepted, but insufficiently studied “principles” enumerated in the well known National Academy of Science’s book “How People Learn” such as “build on prior knowledge”, “integrate conceptual and procedural knowledge”, and “encourage meta-cognition”. Our analysis tools and learning-theory based software will facilitate the creation of detailed computational theories for when, why, and how these principles operate.
PSLC’s LearnLab will enrich the scientific infrastructure by establishing unprecedented collaborations between existing and future laboratory scientists, computer scientists and instructional designers. The Center will provide 50-100 pre-doctoral and post-doctoral students research experience in developing and using LearnLab tools to address challenging learning problems. PSLC will also have broad positive impacts on schools. Experimental manipulations that demonstrate substantial learning gains will be incorporated in LearnLab schools and eventually to many other schools. PSLC researchers in Pittsburgh and worldwide will enhance scientific knowledge about robust human learning by 1) basing both theory development and instructional design on close, detailed, study of long-term student learning, 2) demonstrating the value of online intelligent tutors as research platforms that support learning experiment and theory development, 3) demonstrating a new synergy between use-driven research and theory-testing instructional design.