July 1, 2024
Introduction & Executive Summary
Mathematics is undeniably challenging for students. Global assessments consistently reveal low proficiency, and studies indicate that individual math performance tends to decline as students progress through their academic journey. In this context, math error identification and feedback are crucial. Regular and timely feedback on math errors improves performance, rectifies misconceptions, and nurtures a deeper understanding of mathematical concepts. It enables students to pinpoint areas of weakness and supports learning of correct concepts and procedures. Without such feedback mechanisms, students risk perpetuating errors and developing misguided notions about mathematical principles, hindering their overall academic growth and contributing to disinterest or low math confidence. Thus, integrating robust error identification and feedback strategies into math education is imperative for developing skilled mathematical thinkers.
In March 2024, leading researchers, technologists, and practitioners in K-12 convened for the Math Errors, Interventions, and Opportunities for AI (MEIOAI) Workshop, outlining strategies for identifying, classifying, and addressing math errors and misconceptions. The workshop aimed to facilitate collaboration around AI-driven research and development, as participants discussed various technology and machine learning-driven approaches to classify student thinking and improve math learning outcomes.
The workshop highlighted significant interest in identifying and effectively intervening in math errors, underlining the importance of collaborative efforts across various sectors. The main themes discussed included the development of sophisticated data collection methods to enhance understanding of students’ thought processes during an error incident. Furthermore, the potential of AI and the necessity of considering both student and teacher experiences in data collection and analysis methods were emphasized, suggesting a holistic approach to education that incorporates advanced technology and empathetic pedagogy.
This report details these workshop discussions and summarizes the current work in understanding, identifying, and providing feedback on math errors while recommending future research and development priorities to increase learning outcomes among math students and reverse the declining trend in math performance.
First, the report overviews common math error types, including procedural errors, slips, weakly held beliefs, and misconceptions. It then outlines key themes and trends in the field for appropriate diagnosis and interventions. Finally, this report delves into comprehensive recommendations for math education, emphasizing the design of diagnostic assessments and the utilization of diverse datasets to refine AI algorithms. Additionally, the report explores essential capacity-building strategies, including fostering collaborations for unified taxonomies and enhancing professional development. These sections aim to outline actionable steps for researchers, scientists, and educators to effectively integrate cutting-edge technologies into the fabric of math learning and instruction.
Understanding Math Errors
To provide an optimal learning experience for the student, teachers must understand why the student made the error. A student will need different instructional interventions depending on whether they make an error due to not understanding the underlying principles and ideas of the problem (e.g., conceptual) versus an error caused by not knowing or executing the steps that should be taken in solving the problem (e.g., procedural). Various types of errors exist in math, ranging from slips – when students make mistakes despite knowing the correct procedure – to misconceptions stemming from incorrect conceptualizations that can lead to systematic patterns of errors.
Procedural Error
Procedural knowledge pertains to knowing the steps to solve a problem. A procedural error, for example, may present as students not finding a common denominator when adding and subtracting fractions (Booth, 2011). However, procedural errors do not always indicate a lack of procedural knowledge. Slips, for example, are small mistakes made, such as calculation errors, resulting in an incorrect answer despite the student knowing how to derive the correct answer (Kaufmann et al., 2022).
Conceptual Error
Conceptual understanding involves grasping the underlying principles and relationships (Booth, 2011). For example, a common conceptual error in fractions is students not understanding that a smaller denominator value indicates a larger proportion of the whole. In traditional US education, procedural processes are often taught to be memorized while the conceptual understanding of why the process should be followed is ignored (Crooks and Alibali, 2014).
Weakly Held Beliefs
One dimension of conceptual errors is weakly held belief (Goldin, 2022). Students may make errors due to weakly held beliefs for which the lack of commitment to the idea can encourage irrelevant factors to contribute to their response, such as extra mental effort to determine a problem-solving approach. On the other hand, a student may have a strongly held belief that is incorrect and needs to be addressed.
Misconceptions
Another term in the academic literature for understanding conceptual errors is “misconception.” The concept of misconceptions in mathematics education is intricately defined and studied, revealing the complex ways students understand—or misunderstand—mathematical principles. According to Smith, diSessa, and Rochelle (1994), a misconception is any student conception that consistently leads to systematic errors. These can be due to incorrect, incomplete, or partially formed understandings, or from the misapplication of a rule or concept. Misconceptions typically arise as students try to integrate new information into their pre-existing conceptual frameworks, often leading to persistent errors unless these frameworks are significantly altered (Stafylidou & Vosniadou, 2004).
A notable misconception highlighted in research is the whole number bias, where students mistakenly judge the value of decimals based on the number of digits rather than their actual place value, such as perceiving .355 as larger than .8 because 355 is numerically greater than 8 (Durkin & Rittle-Johnson, 2015; Ni & Zhou, 2005).
Preconception
Additionally, “preconception” describes students’ existing theories, shaped by past experiences, which influence their learning of new concepts. For instance, students often carry forward the idea that addition and multiplication invariably increase values, which becomes problematic when dealing with negative numbers or other exceptions (Karp, Bush, & Dougherty, 2014).
Understanding types of math errors and the student’s underlying understanding of the problem is essential for providing optimal tailored instructional approaches. Further research on the types of math errors is needed as the frequency of these math errors and how to best identify them are not yet understood. AI-driven systems to detect and parse the difference between a lack of procedural vs conceptual knowledge, for example, could aid in improving learning outcomes by increasing the relevance and timeliness of feedback.
The MEIOAI Workshop highlighted the complexity of math errors and the dynamic interaction between prior knowledge, instructional cues, and the learning environment. Participants agreed that student experiences can be better designed to support the development of robust math understanding rather than viewing errors as deficiencies in individual learners. While some errors result from unfinished learning and merely require additional instruction, others result from a deeply entrenched learning gap that requires targeted intervention.
Current Trends for Math Error Identification and Feedback
Researchers, digital learning platform creators, and educators alike are contributing to the evolving landscape of math error identification and feedback. While some stakeholders are leveraging cutting-edge technologies, others are creating novel pedagogical strategies. Overall, however, the field is witnessing a convergence in the approaches for enhancing mathematical comprehension and proficiency, with more similarities than differences. This section will highlight some of those common trends.
Cross-Sector Interest in Practical Solutions
The consensus across various sectors is to focus on understanding the practical implications of math errors. Educational technology professionals, school districts, and education experts are increasingly invested in identifying and addressing these issues. They recognize the significant impact that foundational math understanding has on student learning and future workforce preparedness. By examining how math errors manifest in real-world scenarios, these professionals are looking to improve educational outcomes and enhance the design of learning tools and curricula that can better address and prevent such errors in diverse learning environments.
Data-Driven Personalization and Intervention
One emerging trend is the convergence of data, technology, and AI in classroom tools, which will continue to revolutionize error response and intervention in mathematics education. By harnessing the power of advanced analytics and machine learning algorithms, educators can personalize instruction and support based on individual learning profiles and error patterns. Adaptive learning systems powered by AI can dynamically adjust content delivery and scaffolding in real-time, offering tailored interventions that address students’ unique needs and optimize learning trajectories. Moreover, the integration of data-driven decision-making into instructional practice enables educators to identify systemic trends and areas for improvement, facilitating evidence-based reforms at both the micro and macro levels of instructional practice.
Integration of Research into Curriculum and Professional Development
A concerted effort has been made to incorporate research findings into curriculum development and professional development materials for educators. As the body of research on math error identification and feedback expands, educators will increasingly rely on evidence-based practices to inform instructional design and pedagogical approaches. By embedding these insights into curriculum frameworks and professional development initiatives, educational institutions can cultivate a culture of innovation and continuous improvement, ultimately enhancing the quality of mathematics instruction and student learning outcomes. Additionally, the relationship will be cyclical and the interest from the education community will drive more research and foster the creation of better tools and interventions.
Deepening Understanding through Authentic Student Work Analysis
Annotating and analyzing real student work for math error types will also remain a cornerstone of work in this area. When they have access to real data, educators gain valuable insights into students’ thought processes, misconceptions, and areas of difficulty. This in-depth understanding enables educators to tailor their instructional strategies, feedback, and interventions to better address students’ specific needs. Moreover, the emphasis on authentic student work analysis (i.e. free response not multiple choice) fosters a collaborative learning environment where students are encouraged to reflect on their own reasoning and engage in productive dialogue with their peers and instructors.
Translating the Trends to Discussion
The MEIO-AI workshop showcased how these emerging trends are shaping the landscape of math error identification and feedback. Throughout the workshop sessions, participants delved into the transformative potential of evidence-based practices, authentic assessment methods, and data-driven interventions. By embracing these principles and leveraging technological innovations discussed during the workshop, educators gained valuable insights into how to empower students to become adept problem-solvers. Through collaborative discussions and hands-on activities, participants explored practical strategies for integrating research findings into curriculum development, annotating real student work for error analysis, and harnessing the power of data, technology, and AI for personalized intervention. As a result, the MEIO-AI Workshop reinforced the culture of innovation, personalization, and continuous improvement in mathematics education.
A primary focus was on the development of robust data collection and analysis techniques to effectively pinpoint and remedy math errors, including misconceptions. Participants underscored the potential of multi-modal data collection, such as integrating visual and auditory data with written responses, to provide a more comprehensive understanding of a student’s thought process.
For example, ASSISTments not only works with computer-scored and open-ended math questions but also has a dataset of images of student work available. This would support research on the automated analysis of students’ handwritten math work, addressing a highly identified gap in automated scoring.
Moreover, the workshop facilitated the refinement of a proposed taxonomy for categorizing math misconceptions. This taxonomy, designed to enhance clarity and consistency across educational assessments, represents a pivotal step forward in accurately identifying and addressing learning gaps in mathematics. Additionally, the integration of AI and the thoughtful consideration of both student and teacher experiences were highlighted as essential components in transforming math education to be more responsive and effective. For example, Caroline Hornburg at Virginia Tech mentioned the need for research linking children’s experience of the classroom error climate to their particular misconceptions/errors as well as teachers’ own math identity and particular teaching methods.
Participants also emphasized the need to provide teachers with data in digestible formats and actionable steps to support their instruction. For example, detailed student-level analyses can be overwhelming and uninformative when determining the ideal steps or adjustments to teach a whole class.
Case Studies
iES NAEP Math Automated Scoring Challenge: Competition and Data Release
John Whitmer, while an Impact Fellow at the Institute for Educational Sciences, conducted market research into the use of automated scoring to provide scores to open-ended responses. An open data challenge offered via Challenge.gov conducted in 2023 identified three winners who could automatically score math questions with accuracy similar to humans using advanced natural language processing methods. Implementing automated scoring was demonstrated to have the ability to provide accurate scores, faster, and at a lower cost – while increasing information about respondents that could be used for additional insights beyond the score points that are currently reported. The challenge required that teams not only provide predicted scores, but also submit technical reports that describe the processes and models used to generate those predictions, which identified effective practices and algorithmic approaches (namely, transformer models).
The NAEP Math Automated Scoring Dataset was extracted from the NAEP 2017 & 2019 grades 4 and 8 national and state assessments. The dataset includes over 250,000 student-constructed responses to 10 math questions to provide students with an opportunity to explain their reasoning or the process they used to respond to a mathematics item. NAEP currently assembles teams of human scorers to evaluate millions of student responses to its assessments, a process that is laborious and time-intensive, requiring months for scoring. Questions are also re-used between administrations, providing a perfect opportunity for dramatic cost savings through the re-use of accurate models.
In the competition, competitors ascertained how accurately AI automated scoring models predicted human scores. Because these are math items, the agreement between raters was extremely high (>0.95 QWK), but automated techniques successfully predicted scores on 9 out of 10 items, and analysis for bias by student subpopulation did not uncover any issues of fairness. Notably, the one item that could not be scored automatically had very few correct answers by students, an issue that could be corrected in an operational setting.
These findings indicate a promising direction for NAEP as the new contract for NAEP is currently under development. More information about the challenge results and winning teams is posted in a pre-print study. The dataset used for the challenge has also been approved for researcher use under a restricted use data license from NCES.
Eedi
Eedi is an educational technology company that offers personalized learning solutions to improve math education for students and teachers. Through its platform, Eedi provides interactive resources, assessments, and analytics to help educators tailor instruction to individual student needs, optimize learning outcomes, and foster a deeper understanding of mathematical concepts. By leveraging technology, Eedi aims to empower both students and teachers with the tools and insights necessary to enhance math proficiency and drive academic success.
Eedi identifies misconceptions using high-quality multiple-choice questions (Wang et al., 2020), where each wrong answer reveals the nature of a student’s misunderstanding. Teachers use Eedi’s questions in class with no technology as a means of low-stakes formative assessment (Barton, 2018). They also assign questions on the Eedi platform which allows for bespoke support based on their choice of wrong answer and the misconception it reveals.
Eedi’s machine learning research focuses on the application of techniques to predict and understand the evolution of students’ misconceptions over time, highlighting how different misconceptions can adversely affect future learning. To achieve this they have leveraged Knowledge Graphs, enhanced with Knowledge Tracing and Natural Language Processing to encode temporal and textual components.
EdLight
EdLight’s AI-powered platform seeks to revolutionize middle school math education by instantly revealing the ”why” behind student strategies in handwritten work. Edlight is an organization dedicated to revolutionizing the educational landscape by providing innovative digital solutions to enhance teaching and learning experiences. Through its platform, EdLight offers a comprehensive suite of tools and resources designed to empower educators, engage students, and streamline administrative tasks. From interactive lesson planning and assessment creation to data analysis and reporting, Edlight equips teachers with the tools needed to personalize instruction and monitor student progress effectively. Additionally, its platform facilitates communication and collaboration among teachers, students, and parents, fostering a supportive learning community. By harnessing the power of technology, Edlight strives to transform education and drive positive outcomes for all stakeholders involved.
Through thorough analysis, teachers gain deep understanding at both student and class levels and are empowered to provide targeted support and elevate student success. When teachers upload work into the platform, the programs offer a summary of strategies students attempted and an overview of how successfully students were able to implement any particular strategy, including any misconceptions present in the student’s work (this is currently based on human coding, with the goal of using machine learning techniques to do the coding in the future). The program also provides teachers with suggestions based specifically on the misconceptions their students are exhibiting.
Another component of EdLight’s work involves building a dataset that includes approximately 2 million annotations on student work related to the diagnoses of misconceptions while making this data digestible and actionable for teachers.
Opportunities with AI
There is ample opportunity to further student math learning by leveraging AI-driven systems. One promising avenue is the development of machine learning models designed to detect and categorize student errors, including misconceptions. These models can revolutionize math-specific learning platforms by streamlining the process of identifying and addressing common errors, and improving learning outcomes for students through providing real-time feedback tailored to individual student needs. AI tools would also benefit teachers by reducing the time and effort required, particularly in grading and correction of student’s problem-solving.
AI may also be able to parse the differences between a lack of procedural and conceptual knowledge to identify why a student made an error. This would allow for automated targeted interventions that can identify and provide the support and instruction each student needs based on their current understanding of the math problem. AI technologies such as LLMs can play a pivotal role in identifying linguistic cues associated with math errors, leading to deeper insights into student thinking and more effective instruction. Harnessing the power of AI can aid educators in unlocking new possibilities for enhancing math learning and addressing student errors in a scalable and efficient manner.
However, to fully capitalize on AI’s potential in math learning, it’s crucial to build high-quality datasets, taxonomies, and annotation schemes. AI models and algorithms are only as quality and informative as the data provided.
Assessments
AI can revolutionize grading and the creation of assessments for math error identification in many ways. First, training AI models with in-class, authentic assessments designed to surface errors would allow AI models to learn to evaluate student responses and identify errors accurately. Designing assessments that prompt oral discourse and collaborative work provides AI with an opportunity to capture real-time data on student understanding and math errors. Assessment questions that can differentiate between mathematical typos or slips and genuine errors, as well as using chatbots to prompt clarification and obtain richer data, can enhance the diagnostic capabilities of AI models.
Challenges remain in collecting assessment items that reflect a range of math content and errors, but with continued research and innovation, AI-driven assessment tools have the potential to provide valuable insights into student thinking and support successful learning experiences.
Student Data
There is a vast amount of data on student responses to math assessments, from multiple-choice items to open-ended responses, which can be to train AI models in diagnosing students’ mathematical thinking.
While multiple choice is a common method of assessing math knowledge, they don’t track students’ mental processing, which is needed to identify gaps in their knowledge. Open-ended responses, however, can share details of a student’s reasoning. This data is not often available or accessible and is time-consuming to code. The most informative dataset would be multi-modal, including measures such as open-ended written responses, student/teacher interaction videos, student-drawn figures, eye tracking, teacher instruction methods, etc. This would provide a complete picture of student processing and understanding, linking their understanding and classroom experience to the error. A collection of multi-modal data would best provide AI with nuances of the real world, support the mapping of student thinking, and provide guidance on which measures are most informative.
A central theme of the MEIOAI workshop was the need for robust data collection and analysis methods to effectively assess and address math errors, including misconceptions. Participants noted the promise of multi-modal data collection and stressed the importance of obtaining the right level of detail to effectively identify and address errors. For instance, collecting visual or audio data on oral discourse regarding an error made in addition to the written response would provide greater insights into the student’s mental process and ensure the correct misconception is identified.
Participants also highlighted the importance of student demographic data such as race/ethnicity, gender, language learner status, disability, and socioeconomic status to assess bias and validate fairness in AI algorithms and tools. The fairness of AI systems across all demographics must continue to be considered and assessed. Particularly given that there is existing bias in AI, including in education, attention should be paid to reducing and eliminating bias in future models to provide equitable opportunities for all students, teachers, schools, and districts with AI and technology.
Coding Student Errors
Well-labeled datasets are key to training AI models for accurate math error identification and feedback. Developing expert taxonomies and standardized annotations would provide consistency and enable effective collaboration among researchers and educators. Labeled datasets also allow for the exploration of the breadth and depth of errors, providing insights into the connections between high-level labels of errors and prerequisite concepts based on the dataset labeling chosen. For instance, a dataset on early math could have labels of fractions, whole numbers, and decimals. Within this labeling system, there might be a misconception where students believe bigger decimals mean bigger numbers. By connecting the math concept to the error, teachers can more effectively understand and intervene before errors become ingrained over years. In developing taxonomy, involving practitioners’ perspectives and expertise in the annotation ensures relevant coding that applies to real-world educational settings.
Recommendations
Moving forward, future research and development priorities for the field should focus on:
- Standardizing taxonomies for student error types
- Developing assessments specifically designed for extracting misconceptions to enable more in-depth analysis.
- Leveraging AI (e.g., LLMs, generative AI, multi-modal AI) to automatically detect and provide explanatory feedback on math errors
- Releasing more open-access datasets that reflect diverse populations to ensure resulting AI algorithms are unbiased and focused on the most common errors.
- Investing in capacity building and professional development for educators on the intersection of AI, tech, and math misconceptions(to increase adoption and buy-in
Proposed Taxonomy for Math Misconceptions
There was widespread agreement at the MEIOAI Workshop on the importance of careful consideration in labeling and coding errors, including misconceptions. The framework should accurately differentiate error types (i.e., procedural errors, slips, weakly held beliefs, misconceptions). As experts brought up during the workshop, different error types warrant different instructional interventions. For instance, some students make errors despite knowing the manner or concepts needed to solve a problem. In these instances, there is some aspect of the student’s thinking or understanding that needs to be changed to have a fully correct response. It may be easier to address these kinds of errors as opposed to correcting misconceptions. The taxonomy should also reflect the full range of mathematical content so it can be applicable to various topics and grade levels.
Thoughtful, Intentional Assessment Design
There needs to be a strategic focus on the thoughtful and intentional design of assessments that provide accurate diagnostic assessments of student errors. Designing items exclusively to reveal misconceptions can be done through multiple-choice items with diagnostic distractors that reflect common misconceptions among students or more open-ended questions that allow students to explain their reasoning sufficiently. There can also be assessments providing students with interesting and meaningful problems, ensuring enough questions to identify misconceptions accurately. Collaboration among practitioners, researchers, and scientists is needed. Researchers with experience in psychometrics and assessment design can focus on item validity, educators can provide insight and practical evidence of what students are struggling with, and ed-tech platforms applying these items on their platforms can get real data and analytics from students and teachers. The field should also leverage open-source communities to share these assessments widely so that there is a feedback loop to ensure these assessments remain effective and relevant
Assessments should also provide students with relevant and applicable data. For example, assessments should include authentic questions that can demonstrate the importance and use of math in the real world to engage student interest and increase the perceived value of learning math. Addressing student anxiety in math and normalizing the difficulty of math problems is also crucial. Normalizing the difficulty of math and the widespread misconceptions of math can reduce student stress and anxiety, improving the ability to take in information and correct their misconceptions/errors.
More Diverse, Open-Access Datasets
The field also needs to focus on publishing datasets that can advance the work of misconceptions and ensure these datasets are diverse and represent various regions, cultures, backgrounds, grade levels, and academic skills. Open-access datasets will support the work of the research and machine learning communities, creating new benchmarks to train different AI-powered solutions for detecting error-prone thinking. A demographically diverse or representative dataset can also ensure resulting algorithms won’t be biased against diverse populations, particularly historically marginalized populations. Multidisciplinary collaboration in data collection and release is also key here. Once these datasets are shared through open-access platforms, they will facilitate widespread research use.
Leveraging AI
More collaboration is needed among math cognition experts, academics, and researchers with data scientists, computer scientists, and technologists to leverage the latest advancements in AI for math error identification and feedback. The AI landscape is evolving rapidly and improving quickly, and applying these technologies to educational tools that can understand student thinking and diagnose student errors at scale is quite promising. This includes applications in natural language processing with text data as well as within multimodal (audio, speech, video) AI data inputs and outputs. Research and development can focus not only on leveraging foundational pre-trained models for this domain but also on how to integrate these AI models into learning platforms, whether as a single model or as a system of distinct AI models working together.
Additional opportunities for leveraging AI to build tools that could support math error identification and feedback include:
- Personalized tutoring chatbots leveraging generative AI to identify and provide feedback on students’ frequently occurring math errors
- Learning platforms that facilitate adaptive testing based on the student’s past errors
- Personalized learning trajectories based on a student’s current understanding to correct their common errors and guide their development of math concepts
- Personalized learning platforms designed to promote learning through identifying and increasing student confidence in math
- Instructional support tools that provide realistic common responses and errors to a problem or lesson, identifying common ways of student thinking to assist teachers in anticipating errors and developing instructional strategies
Additional opportunities for using AI to address pertinent research questions in the education community include:
- Investigating the link between student classroom error climate (i.e., student reaction to errors) and math understanding, which would yield new findings on the relationship between student perception of error and confidence in math performance.
- Modeling the relationship between student errors and teacher math identity, to assess instruction and teacher influence on student errors.
Capacity Building
More efforts are needed to build capacity in this community. One simple approach is facilitating more opportunities like the MEIOAI Workshop and designing conferences and community forums that can get key leaders across research, academia, school leadership, digital learning, and ed-tech innovation in the same setting to share ideas, identify best practices, and explore opportunities for collaboration to advance this work. Building confidence and buy-in at the intersection of AI, educational research, and academic training is also important. A key strategy may be in the inclusion of teachers in these research and development projects, as well as providing them ongoing support and resources, like professional development training, for them to understand how to use AI effectively for diagnosing student errors or executing interventions based on the nature and depth of the math errors.
Conclusion
The MEIOAI Workshop highlighted the pressing need to integrate AI technologies into math education to address and correct student errors effectively. Bringing together math cognition experts, academics, researchers, data scientists, computer scientists, and technologists is essential to harness AI advancements for better error identification and feedback.
In summary, integrating AI into math education has the potential to revolutionize error identification and feedback, empowering students to become confident problem-solvers and improving their overall mathematical proficiency.
__________________________________________________________________________________________________________________________________
– This article was written by Jules King, L Burleigh, Perpetual Baffour, Scott Crossley, Bethany Rittle-Johnson, Kelley Durkin, Rebecca Adler, Meg Benner, and Ulrich Boser.
