Home | Research Statement | Current Papers | Curriculum Vitae
Basic Research: Intuitive Strategies and Formal Representation Learning
My basic research has focused on the role of informal, invented, or intuitive knowledge in the learning of mathematical problem solving and reasoning. To the everyday person (and to many researchers), the well-structured and rule-bound surface of mathematical reasoning gives the impression that the underlying thinking process has the same structure and rules. My basic research debunks this tempting analogy and, furthermore, precisely indicates what the rules of mathematical thinking are if they are not the rules of mathematics itself.
Intuitive Knowledge Behind Geometry Theorem Proving
For instance, in Koedinger and Anderson (1990) we described a cognitive model of how good problem solvers are able to make intuitive leaps of inference in discovering and creating proofs of geometry theorems. Previous theories hypothesized that experts prove theorems by searching through the space of formal rules of the domain guided by heuristics but fundamentally using the definitions, postulates, and theorems that appear in geometry textbooks. A careful analysis of verbal reports of human experts revealed something quite different. When planning a proof, experts do not use the textbook rules. Instead they make leaps of inference to first sketch out the key steps of the proof, and only then use the textbook rules to fill in the details. We discovered that this abstract planning was enabled by concrete, perceptually-based knowledge representations we called diagram configuration schemas. These schemas were used to create a running cognitive model that provided 1) an accurate empirical account of expert behavior including quantitative predictions of the steps they skip in planning, 2) a geometry expert system that was much more powerful and efficient than previous cognitive models, and 3) the core student model for a new intelligent tutoring system called ANGLE. More generally, this work contributed to our understanding of how diagrammatic reasoning can aid reasoning and learning (Koedinger, 1992a), provided a unified explanation of number of established characteristics of expertise including perceptual chunking and working forward bias (Koedinger & Anderson, 1990), and posed a challenge for learning theories given the discovery of an alternative perceptually-based representation of expertise that does not clearly follow from existing learning algorithms (Koedinger, 1992b).
Designing Novel Interface Representations to Facilitate Thinking and Learning
One of the themes of the geometry work was that much of the thinking process behind competent performance is under the surface and not revealed in the actions performed, nor made explicit in textbooks or typical instruction. To address this “implicit planning” issue through educational technology, we recommended the design of computer interface notations to reify or make visible otherwise implicit thinking steps (Koedinger, 1991; Koedinger & Anderson, 1993a). This “plan reification” approach has worked well in some cases, like the flow proof representation of search paths in the ANGLE geometry tutor, but not in others, like the problem decomposition interface in an early equation solving tutor (Anderson, Corbett, Koedinger & Pelletier, 1995). In the latter case, the resulting pedagogical interface was not similar enough to the target performance interface, namely, solving equations on paper and thus, did not facilitate transfer of learning. The design lesson is that a good pedagogical interface should help reveal the thinking process, but not at the expense of obscuring the similarity to the desired performance interface.
Generalizing Cognitively-Based Instructional Design Results
To extend and generalize the geometry work, I began research in the algebra domain focusing on word problem solving. Like the process of constructing a proof, the steps in translating problem situations to algebraic symbols are not explicitly addressed in typical instruction. As an alternative to using costly verbal protocol analysis to identify these implicit planning steps, we applied the theory of acquisition that had come from the geometry research to infer what these steps might be. Given the prior discovery of the critical role of inductively-derived intuitions in geometry, we hypothesized that students’ acquisition of formal algebra symbols could be enhanced by encouraging them to induce algebraic sentences from concrete instances of arithmetic procedures. We contrasted tutorial instruction using textbook problems like the one below with “inductive support” instruction in which we placed the concrete instance questions (#2 and 3) in front of the symbolization question (#1).
Drane & Route Plumbing Co. charges $42 per hour plus $35 for the service call.In an experiment comparing early versions of the Pump Algebra Tutor (described below), we found that students using the inductive support version learned significantly more than students using the textbook version (Koedinger & Anderson, in press). This study and the geometry work provide evidence for the inductive support hypothesis: Formal knowledge grows out of prior inductive experience which provides a semantic foundation in the form of back-up strategies for retrieving, checking, or rederiving formal knowledge.
1. Create a variable for the number of hours the company works. Then, write an expression for the number of dollars you must pay them.
2. How much you would pay for a 3 hour service call?
3. What will the bill be for 4.5 hours?
4. Find the number of hours worked when you know the bill came out to $140.
One current off-shoot of this work is a specific research focus on the inductive process itself. This work, with psychology graduate student Lisa Haverty, focuses on modeling what students know and can learn about inducing rules from data (Haverty, Koedinger, Klahr, & Alibali, in press).
Intuitive Strategies in Algebra and the Multiple Strategy Effect
Are the informal strategies we observed of high school students a fleeting and temporary developmental transition in students’ algebra learning or do such strategies stay in students’ repertoire as they acquire formal strategies? To investigate this question, I began a collaboration with postdoc Hermi Tabachneck in which we analyzed verbal protocols of CMU college students solving algebra word problems. We were surprised at the extent to which college students were using strategies other than the formal algebra equation strategy of translate and solve. In addition to this strategy, students used a variety of informal back-up strategies, for example, working with concrete instances by guessing at values for unknowns. In an analysis of the strategies used, we found that students were much more successful on problems where multiple strategies were used (80% correct) than on problems were only a single strategy was used (40%; Koedinger & Tabachneck, 1994; Tabachneck, Koedinger, & Nathan, 1994). We formulated a detailed account of these alternative strategies and provided a theoretical explanation of the multiple strategy effect by identifying how different strategies address different cognitive demands (familiarity, efficiency, memory load) and thus have complementary benefits (Tabachneck, Koedinger, & Nathan, 1995). Thus, informal strategies are not only important for learning, but are also an important part of students’ lifelong mathematical tool kit.
A New Methodology for Cognitive Model Development: Difficulty Factors Assessment
Extensive experience with building cognitive models through the time-consuming and somewhat subjective process of protocol analysis began to make me seek a complementary methodology. I wanted a methodology that: 1) could more quickly yield results that could inform instructional design, 2) provide empirical data about the nature of knowledge and performance that is less suspect to subjective protocol coding judgments, and 3) not only reveal the thinking process, but also indicate something about learning.
I developed a methodology called "Difficulty Factors Assessment" which we has now used in a number of published (Aleven, Koedinger, Sinclair, & Snyder, 1998; Heffernan & Koedinger, 1997, 1998; Koedinger & MacLaren, 1997; Koedinger & Tabachneck, 1995; Tabachneck, Koedinger, & Nathan, 1995; Verzoni & Koedinger, 1997) and unpublished studies. In Difficulty Factors Assessment (DFA), we use theory and task analysis to generate hypotheses about the likely factors that cause student difficulties and then use these factors to systematically generate a pool of problems. By assessing performance differences on pairs of problems that vary by one factor only, we can identify what knowledge elements are needed in a model in order to adequately decompose the thinking and learning processes of students. The first DFA studies addressed the domain of early algebra problem solving. Based on our discovery of the important role of unschooled, intuitive strategies in geometry proof and algebra symbolization, we hypothesized that students may be more likely to retrieve or create invented strategies when problems are presented in a familiar context, like a word problem (e.g., “Ted works 6 hours and gets $66 in tips. If he made 81.90, what’s his hourly wage?”), than when presented in an analogous equation (e.g., x * 6 + 66 = 81.90). In fact, contrary to the predictions of mathematics teachers and educators (Nathan, Koedinger, & Tabachneck, 1997; submitted), we demonstrated (and have repeatedly replicated) that high school students are better at these simple algebra word problems than the corresponding equations (Koedinger & MacLaren, 1997; Koedinger & Tabachneck, 1995; Verzoni & Koedinger, 1997). Furthermore, it is not so much the problem situation that facilitates performance (the prediction of situated cognition theory), but the relative familiarity of English language descriptions over symbolic ones -- problems in words without a situation (e.g., “Starting with some number, if I multiply it by 6 and then add 66, I get 81.9. What number did I start with?”) are also easier than the analogous equations.
Expert Blindspot and the Risks of Informal Design
The contrast between educator predictions and the students’ actual difficulties is an instance of a more general phenomenon I call “expert blindspot”. As we develop expertise, for instance with algebraic symbols, we lose the ability to accurately introspect on problem difficulty from a student’s perspective. Consistent with this notion, we found that teachers with the most experience (high school math teachers) are the most likely to make the incorrect prediction that students would find the given word problems more difficult than the analogous equations. Middle school and elementary teachers are successively less likely to make this incorrect prediction, though they still do so in large numbers (Nathan, Koedinger, & Tabachneck, submitted).
I have often observed the expert blindspot phenomenon when educators have informally evaluated our intelligent tutoring systems. In some cases, they comment on how inflexible the tutor is, for instance, not allowing them to go on before fixing an error. Or, in the case of my ANGLE geometry tutor, some have been impressed with its flexibility. However, empirical studies have shown that what is inflexibility to an expert is support for the novice (e.g., students learn more efficiently with immediate feedback) and what is flexibility for the expert can be confusing for the student (e.g., students showed greater signs of floundering in ANGLE than in the earlier, less flexible, geometry proof tutor, Koedinger & Anderson, 1993a). More generally, multimedia, animations, and educational games have appeal because they often appear to make difficult ideas clear and lively. However, this clarity is much more apparent to an expert who already knows the difficult ideas being illustrated. Empirical studies have shown, for instance, that students do not always learn what we expect from games purported to be educational (e.g., Miller, Lehman, & Koedinger, in press).
The expert blindspot phenomenon has important implications not only for educational technology and how it is evaluated (mostly it is not -- see the November, 1997 Consumer Reports for a pathetic example), but also for software evaluation more generally. Software designers who are expert programmers or experts in the end-user domain are susceptible to expert blindspot and are likely to create applications that are intuitive and easy for them to use, but unintuitive and hard for learners who are novices in the interface or the content domain. Because of expert blindspot, seat-of-the-pants interface design is not likely to work and empirical studies of users are critical.
Using Difficulty Factors Assessment to Create Cognitive Models
The evidence for expert blindspot comes from comparing expert’s predictions with real performance data, in our case, data generated by statistical analyses of main effects and interactions in Difficulty Factors Assessments (DFAs). A second stage of DFA analysis involves coding students’ written solutions to identify common strategies and errors. Like the college students, we found students using a number of intuitive strategies in addition to the normative algebra strategy. We also found a pattern of errors that suggested students’ major difficulty with symbolic equations is in the early stages of comprehending the meaning of the equations. Using the ACT-R theory and production rule language, we built a model of student thinking that captured the intuitive and implicit nature of student strategies and errors (Koedinger & MacLaren, 1997). The model provides an excellent fit to the student process data, accounting for 90% of the variance in average student performance for major strategies and errors across a variety of problem types. The model predicts that students’ poorer performance on symbolic problems than word problems is not a function of some necessary cognitive prerequisite (e.g., verbal knowledge must proceed symbolic knowledge), but rather it is a question of exposure and practice. According to the model, students are more familiar with verbal descriptions of quantitative relationships because they have more experience with them than with symbolic descriptions. Thus, the model predicts that if students’ experience with symbolic descriptions is as frequent as their experience with verbal descriptions of quantities, then there should be no performance difference. Indeed, when we gave the problem-solving DFA to Russian students who start using symbolic sentences in second grade (unlike US students who start seeing algebraic symbols in middle school), the performance difference all but disappeared.
In addition to providing a good to fit to the overall pattern of students’ strategies and errors, our ACT-R model also accounts for 68%-90% of the variance in the strategy and error frequency at six different performance levels. These performance levels and the cognitive process differences between them are the basis for a developmental theory. Testing and extending this developmental theory of algebraic representation learning is a major topic of a four-year McDonnell Foundation project (with Mitch Nathan, a University of Colorado Math Educator, and Martha Alibali, a CMU Developmental Psychologist) that just started in June.
Future Work: Automating Difficulty Factors Assessment
Another element of that project involves the automation of the DFA methodology in software for cognitively-based adaptive testing. We have created a prototype “Cognitive Assessor” that we have been testing using the ACT-R model as a simulated student. The model is parameterized at a particular developmental level and the Assessor’s job is to determine this level by having the simulated student solve specifically chosen problems. In early tests, we have been contrasting a Bayesian estimation algorithm with a new “Problem Graph” algorithm of our own design which, currently, is doing a better job of diagnosis.
Learning Representational Languages
The results of these and other DFA studies have led to the idea of thinking about algebra as a language. Learning the language of algebra is much like learning any other natural or artificial language. Learners must acquire (at least implicitly) the syntax and semantics of the algebra language as well as separate classes of skills for both comprehending and producing sentences in this language. Just as going from simple clauses to complete sentences is a major step for children learning their first natural language, in learning algebra students must acquire the (implicit) syntax not only for simple clauses (e.g., 40x) but also for “embedded clauses” (e.g., 800 - 40x). With DFA studies we have identified a clear developmental transition between these states (Heffernan & Koedinger, 1997; Heffernan & Koedinger, submitted). Like natural languages, many artificial languages have a transformational syntax and semantics (the structure and meaning of manipulations between sentences in the language) as well as a representational syntax and semantics (the structure within sentences and their meaning, i.e., the referential mapping between language elements and the objects and events represented). In algebra, the transformational syntax and semantics are the rules for manipulating equations and their meaning-giving justifications. Most prior cognitive research as well as the major focus of traditional algebra instruction has been on the transformational issues, namely, helping students learn to solve equations. To the extent there has been cognitive research on algebra as a representational language, it has focused primarily on word problem solving and, in particular, on identifying and modeling the skills for comprehending the English sentences in word problems. The DFA studies have shown, in contrast, that the learning hurdle for algebra students is not so much comprehending the English sentences, but producing Algebra sentences.
Future Work: The Quantity Animator as a Pragmatic Context for Algebra Language Learning
If learning a natural language is aided by the pragmatic context in which children practice that language, then perhaps learning the language of algebra can be aided by creating a pragmatic context in which students can practice that language. In particular, that context should be one in which the algebraic sentences that a student produces has consequences that the student can easily interpret. In such a context, a student should be able to engage in the following thought processes: 1) Does what I say lead to the consequence I expect? 2) If not, what consequence does it lead to? 3) How can I change what I'm saying to lead to the desired consequence?
We have begun to develop a software tool that can provide such a context, in the form of an animation, for learning the syntax and semantics of algebra. In the "Quantity Animator" students are working from a story problem and begin by choosing two animation panels, one to represent the X quantity (e.g., a clock to represent time in minutes) and one to represent the Y quantity (e.g., a boat on a lake to represent the distance boat is from the dock). Next they write an algebraic sentence for the Y quantity in terms of X (e.g., 800 - 40X) and then run the animation. The animation is driven by whatever expression is entered. Having read the problem statement, the student has an expectation for what should happen in the animation. If the animation violates these expectations (e.g., the boat moves in the wrong direction because 40X was entered), they have a basis for (a) understanding the correct meaning of what they said, (b) adjusting their sentence to achieve the desired consequence, and (c) learning something new about the syntax or semantics of algebra.
We have created a prototype Quantity Animator and subjected it to a number of formative evaluations (executed in part by HCI Masters students). A redesign has been proposed that will be implemented and tested this fall.
Designing Intelligent Math Tutors for Classroom Integration
Basic research is important, but if I cannot see my research brought to bear on an important human problem, I will not be satisfied. The problem driving my research career has been improving mathematical and technical education. The importance of this problem is highlighted by two recent news events. The results of the Third International Mathematics and Science Study (TIMSS) recently came out (the 12th grade results in February, 1998) and US performance was poor, significantly below many other countries in 8th and 12th grade math and science and below the international average in most categories. Also in the news recently have been reports of shortages in employees with technical skills, particularly programmers. Referring to the TIMSS results, the Chairman of the National Alliance of Business said, “For the business community, these results are chilling ... [they] indicate that our students are at a disadvantage in terms of the skills required to meet the challenges they will confront”. Our poor international showing is not a consequence of declining educational effectiveness in the US, in fact, math achievement on the National Assessment of Educational Progress (NAEP) has gone up in the last 20 years. Instead, we are not keeping up with other countries and we have been particularly unsuccessful in making mathematics accessible to the cultural and socioeconomic diversity of our population.
The key vehicle for applying my basic research results to improving mathematics education has been intelligent tutoring systems. However, the basic research results have also influenced the instruction process more generally including aiding the design of supporting curriculum materials, assessments, and methods for teacher training. Individual human tutors have been shown to increase student achievement by as much as two standard deviations over traditional instruction. In evaluations of our intelligent tutors we have found about a one standard deviation improvement over traditional instruction.
Field Studies with ANGLE: The Need for Curriculum Integration
My initial venture into real classrooms made the importance of curriculum integration efforts plainly clear, both in a quantitative statistical way and in a qualitative personal way. This classroom field study involved the ANGLE tutor for geometry proof, discussed above, in which we designed the interface notations to reify or make more concrete to students the inductive planning strategies of good problem solvers. ANGLE is also the system in which we observed (and perhaps were guilty of) expert’s blindspot -- experts who tried ANGLE frequently commented positively on how it was more flexible than the previous geometry proof tutor. As noted above, this increased flexibility, or from the student perspective, reduced instructional support, made the learning job of the student more difficult. It also appears, in retrospect, that this flexibility put greater onus on teachers to fill in the needed support.
At the time of this study, the Langley High School had responded to the National Council of Teacher’s Mathematics (NCTM) Standards recommendation to reduce the emphasis on proof in geometry. Their textbook focused as much on inductive reasoning as deductive reasoning. On one hand this change fit well with ANGLE’s emphasis on inductive reasoning as the basis for proof planning knowledge. However, on the other hand, it required a major integration effort to justify extensive use of ANGLE and make appropriate connections with the reform textbook. A teacher on our project team helped with this effort. We created “truth judgment” exercises, which unlike the unnatural traditional approach of asking students to prove well-proven theorems, required students to determine whether a given conjecture is true or false using either inductive or deductive methods.
During the two month classroom field study (reported in Koedinger & Anderson, 1993b), the project teacher taught two experimental classes with ANGLE and two control classes without ANGLE. Two other teachers each taught one experimental and one control class. On a final exam of proof skill, the project teacher’s experimental classes dramatically outperformed all other classes scoring 65% on average while his control classes and the other teachers’ experimental and control classes all averaged around 40% -- a one standard deviation effect size. ANGLE was dramatically effective in the hands of a teacher experienced both in the details of its use and in the subtleties of the curriculum integration plan (as evidence of the latter, his experimental and control classes scored significantly better on a final Truth Judgment test than the classes of the other teachers). The experiment also demonstrated the need for more extensive teacher training and a more structured tutor that was more clearly integrated with the classroom.
During this same period of time, I taught two geometry classes myself. I experienced first hand the challenges urban teachers face, not only in technology integration, but more generally. It was an eye opening experience that was more mentally and emotionally draining than anything I have experienced before or since. Both this teaching experience and the experimental results made clear that we needed to play closer attention to teachers needs if our tutors were going to be generally effective.
Field Studies with PAT: The First Widespread Success of Intelligent Tutors in the Classroom
In response, in the subsequent PAT (Pump Algebra Tutor) project took a “client-centered” design approach whereby we took the approach that we could provide the technology and psychological research, but we would get the content guidance from the NCTM standards, teachers, and curriculum writers. We teamed up with Bill Hadley, a teacher writing a new real-world, common-sense-based algebra curriculum and text. We developed PAT together with the curriculum so that they were tightly integrated from the start. We also began an intensive teacher training program in which teachers attend a week-long workshop in August that addresses 1) NCTM Standards recommendations for increased problem solving, multiple representation use, and mathematical communication, 2) new modes of teaching like cooperative learning groups, 3) details of the curriculum materials and integration with PAT, and 4) lots of hands-on experience using PAT. The first major field study with PAT involved three Pittsburgh public schools and over 500 students. The study demonstrated dramatic effects of the combination of the curriculum and PAT relative to traditional algebra instruction. We assessed students both on the targeted NCTM objectives of problem solving and representation use as well as the basic skill objectives of traditional courses and standardized tests. Students in experimental classes outperformed control classes by 100% on assessments of the targeted problem solving and multiple representations objectives (a one standard deviation effect). They were also 15% better on basic skills as measured by standardized test items from the Iowa and math SAT (Koedinger, Anderson, Hadley, & Mark, 1995; 1997). These results have been replicated now in city schools in Pittsburgh and Milwaukee showing an improvement range of 50-100% on the new standards assessments and 15-25% on the standardized test items.
With a Department of Education-FIPSE grant, I started a project to adapt and field test PAT in the context of college-level remedial math courses. Although the high school field studies demonstrated the effectiveness of the complete PAT solution, including the curriculum materials and reform teaching practices, they did not isolate which features of the complete solution are contributing to the effect. The colleges initially adopted PAT as an add-on to their traditional course. However, across successive semesters they began to increasingly integrate and incorporate PAT principles into the curriculum and classroom practices. The consequences of PAT use, subsequent improvements in integration, and reforms in the classroom curriculum and teaching practices led to dramatic and consistent increases in student problem solving ability across multiple semesters (Koedinger & Sueker, 1996; Koedinger & Sueker, submitted). At the University of Pittsburgh site, for instance, final exam problem solving performance in control classes averaged 35% but was 45% in the experimental classes after the PAT add-on in the first semester and then showed successive increases, 65%, 67%, 70%, 80%, in subsequent semesters as the reforms took shape and hold.
Field studies do not easily afford tightly controlled comparisons of all variables of interest, however, the natural variability of implementation conditions across the semesters at the three sites allowed for a number of semi-controlled contrasts. These contrasts provided some evidence for the independent value added of each of the following features of the PAT curriculum and tutor: 1) PAT’s intelligent assistance providing a one-on-one tutor for each student, 2) a focus on real world problem solving and learning by doing, 3) a tight integration of the technology and classroom activities, and 4) less lecture time and more group projects.
PAT is now in use in some 40 urban and suburban high schools, middle schools and colleges. New schools are now calling regularly about participating in the program which now costs $25,000 per school for the complete training, curriculum and software package. Grants from Darpa and the Department of Defense Education Administration (DoDEA) have further supported the PAT development and dissemination effort including current support for a cross-platform and web enabling port of the PAT interface to Java.
The PACT Center Project
Based on the success of PAT, John Anderson, Albert Corbett, and I formed the Pittsburgh Advanced Cognitive Tutor (PACT) Center funded by a group of local foundations. The goal is to create curriculum materials, training, and cognitive tutors for three years of high school math (geometry and algebra II in addition to the current algebra I course) and demonstrate a standard deviation increase in high school math achievement. On the SAT, a standard deviation improvement would mean a score increase of 100 points.
In the PACT project I have focused on the geometry course and tutor research and development. Following the prescription that good instruction should build on students’ prior strategies and knowledge, we have rearranged the standard geometry curriculum so that the more grounded and applied topics of Area and Pythagorean Theorem are addressed first (these topics are also the most important prerequisites for college as reflected in their prevalence on the SAT test). These topics are addressed using spreadsheet and algebra equation tools like those used in PAT and thus, the course also builds on and reinforces students’ prior algebraic knowledge.
In addition to the traditional focus on normative procedures, formulas, and theorems, the PACT Geometry Tutor also provides students with opportunities to learn and employ alternative back-up strategies to encourage deeper understanding of the content. The goal is to help students construct more rich, interconnected knowledge that leads to more robust performance and better retention and transfer. Back-up strategies are as much strategies for learning as they are strategies for problem solving -- they provide students with a chance to solve problems when expert knowledge is not available and, in turn, an opportunity to learn more expert strategies by deliberately reflecting on or implicitly compiling and tuning their successful experience.
For example, one powerful back-up strategy when facing a difficult problem you do not know how to solve is to break down the problem into simpler problems you do know how to solve. The PACT Geometry Tutor supports this problem decomposition strategy. To reify the implicit planning underlying this strategy we did not have to create an unnatural pedagogical interface (as in the original algebra equation solving tutor mentioned above). Instead we were able to make natural use of the performance interface, a spreadsheet, to do so. If a student is having difficulty with a hard problem, like finding the area of scrap metal after a circle is cut from a square, the tutor suggests that the student create simpler problems by adding columns for the area of the square and the area of the circle.
Future Work: Natural Interfaces for Plan Reification
Using the spreadsheet to reify problem decomposition provides a model solution to the design dilemma, described above, namely, to create interface elements that provide pedagogical support without making these a new learning hurdle that does not transfer to targeted performance interfaces. In a new NSF LIS grant to create a statistics tutor (led by Marsha Lovett, CMU Center for Innovation in Learning and including CMU Statistics faculty), we are employing this design approach to create an interface that naturally reifies and supports the planning process in exploratory data analysis. As an alternative to creating a goal tree planning interface (Lovett’s original idea), we will explore the use of a semi-structured word processor in which students indicate their plans while writing the introduction and methods sections of their data analysis report. In this way, a more natural (and transfer appropriate) performance environment can be appropriated for the pedagogical purposes of plan reification.
Plug-in Tutor Agents for Data Analysis, Experimentation , and Argumentation
Consistent with the idea that the interfaces students use in our tutors are an important part of the learning experience, we have engaged in a technical effort to create a plug-in tutor agent architecture (Ritter & Koedinger, 1996; 1997) whereby we can combine our tutors in a component-based fashion with the best of existing performance interfaces, like Microsoft Excel, and pedagogical interfaces, like simulations and argument construction tools created by other educational technologists (Koedinger, Suthers, & Forbus, 1998). To be effective educational components must achieve some level of semantic interoperatibility, for instance, the simulation, argument tool, and tutor must share, at least in effect, a common definition of experimental trial. A key insight of this work is that this shared semantics can be achieved without a top-down standard that enforces a common representation (and eliminates the advantages of allowing different components to optimize the representation for different purposes), but rather through the use of representation translator components. In Koedinger, Suthers, & Forbus (1998) we demonstrated the combination of three independently developed educational components interacting to support a student in scientific experimentation and argumentation. Picking up a thread from prior work on scientific reasoning (Streibel, Stewart, Koedinger, Collins, & Junck, 1987) and argumentation processes (Koedinger, in press), the tutor agent I contributed to this project contains a prototype cognitive model for domain independent experimentation and argumentation strategies.
Future Work: Tutoring for Deep Knowledge
One of the recommendations of the Third International
Mathematics and Science Study, mentioned above, is that, like other countries,
we ought to be emphasizing fewer topics and addressing them more deeply.
Comparative studies of textbooks and curriculum content suggests that our
curriculum is “a mile wide and an inch deep”. However, must of the
public and research dialogue regarding this issue is quite vague about
what it means to have a deep understanding. A tutor design and experimentation
project under way with postdoc Vincent Aleven is explicitly addressed at
this issue. We have identified evidence of shallow reasoning strategies
in geometry novices and have operationalized the notion of deeper learning
in a cognitive model for searching and interpreting resources, in this
case, a geometric rule glossary. This learning strategy is implemented
in a variant of the PACT Geometry Tutor (Aleven, Koedinger, Sinclair, &
Snyder, 1998) and is currently being experimentally tested against the
prior version of the tutor.
Related work with psychology graduate student Adisack
Nhouyvanisvong is investigating the role of more open-ended problems in
facilitating deeper learning (Nhouyvanisvong & Koedinger, 1998).
Future Work: Modeling Human Tutors and Socratic Dialogues
In collaboration with the University of Pittsburgh we have created a new five year, NSF funded research center, called CIRCLE, targeted at creating third generation intelligent tutoring systems that are capable of engaging students in the kind of knowledge construction dialogues we see in good human tutors. Empirically, the center goals are to better understand the nature of human tutoring and, as a consequence, to create better computer tutors that go beyond our current one standard deviation level of effectiveness and get closer to (or beyond) the two standard deviation effectiveness of human tutors. Technically, the center goals are to advance Natural Language processing and dialog planning techniques and to demonstrate their practical feasibility in making intelligent tutors more human like and/or more effective. As part of that project, computer science graduate student Neil Heffernan and I are engaged in study of human tutors of algebra, creation of models of their best tutorial strategies, experimental tests of these strategies, and implementation and evaluation of them in an intelligent tutor.