As the demand for better AI tutors grows, researchers and developers face a critical challenge: how can they determine whether their platforms actually help students learn before deploying them into classrooms?
Testing AI tutors directly with students is necessary, but it shouldn’t be the first or only line of evaluation. This approach is expensive, time-consuming, and increasingly difficult to conduct. What’s more, there is growing concern that classrooms should not serve as testing grounds for educational software, particularly when the technology is still being developed.
At the same time, securing access to students for research can take months or even years. These barriers can slow development and make it harder for researchers and developers to identify and improve promising AI tutors before they reach classrooms.
To overcome this challenge, researchers are exploring new approaches that allow AI tutors to be evaluated in simulated, realistic learning environments before they reach students. Rather than waiting months to secure research access and conduct live testing, developers could test their platforms with thousands of simulated students in a single day, gathering high-quality feedback and insights in a fraction of the time and at a fraction of the cost.
Teaching AI to Learn: A Foundation for Simulated Students
For decades, researchers have explored how simulated students can be used to support human learning. This work has focused on creating teachable AI agents that have been programmed to behave like learners — demonstrating misconceptions, responding to instruction, and revealing gaps in understanding.
One early example of this approach is Betty’s Brain, a learning environment developed in 2005 to help students deepen their understanding of science concepts by teaching a virtual student named Betty. As students build causal models, explain concepts, and correct Betty’s mistakes, they must constantly evaluate her understanding and adapt their own explanations. By modeling a learner that can make errors and respond to feedback, Betty’s Brain demonstrated how AI agents could create meaningful learning interactions.
For decades, researchers have explored how simulated students can be used to support human learning. This work has focused on creating teachable AI agents that have been programmed to behave like learners — demonstrating misconceptions, responding to instruction, and revealing gaps in understanding.
Intelligent tutoring systems (ITS) have taken a related approach, using explicit models of student knowledge to adapt instruction. Researchers at Carnegie Mellon University first developed the Cognitive Tutor in the 1990s, drawing on research in artificial intelligence and cognitive psychology to model how students solve problems and provide targeted guidance as they work. Carnegie Learning was later founded to bring this research into schools. This work demonstrated how AI systems could use models of student knowledge to adapt instruction to individual learners.
More recently, projects like ALTER-Math, supported by the Learning Engineering Virtual Institute (LEVI), have merged these ideas with generative AI. Rather than simply receiving explanations from an AI system, middle school students strengthen their mathematics understanding by teaching an AI bot that has been programmed to act — and make mistakes — like a middle schooler. As students diagnose the AI’s incorrect reasoning and misconceptions, and explain how to solve multi-step problems, they practice organizing and applying their own knowledge.
While these systems were designed to help students learn by teaching, they established that AI agents can be designed to behave like learners. They can be programmed to make mistakes, demonstrate incomplete understanding, and respond to instruction. Today, researchers are programming simulated students to help them troubleshoot and refine learning platforms before they reach real classrooms.
While these systems were designed to help students learn by teaching, they established that AI agents can be designed to behave like learners. They can be programmed to make mistakes, demonstrate incomplete understanding, and respond to instruction.
Shifting from Outputs to Outcomes
By using simulated students to test AI tutors in realistic learning interactions, developers can evaluate AI tutors in a classroom-like context without exposing students to unrefined technologies. Because learning depends on factors like subject area, grade level, and prior knowledge, AI tutors require evaluations that reflect the specific learners and instructional goals they are designed to support. Traditional metrics may tell researchers whether an AI tutor provides accurate answers, responds appropriately to a prompt, or adheres to surface-level instructional features, but they do not necessarily show whether the interaction helps a student learn. A robust benchmark, like MathTutorBench, can evaluate the system’s pedagogical abilities, including its capacity to diagnose specific student errors and generate appropriate scaffolding rather than simply assessing whether it can provide correct answers.
Simulated students recreate the back-and-forth interactions that can define authentic learning. In a recent study, researchers found that simulated students could consistently reflect different levels of understanding, generate realistic misconceptions, and distinguish between helpful and unhelpful tutoring approaches. These capabilities allow developers to evaluate whether an AI tutor can infer a student’s skills and knowledge and adapt its approach to support the student’s learning.
As the AI tutor works with the simulated student, researchers can evaluate how effectively the tutor supports learning. Understanding changes in a student’s skills and knowledge is a key component of a successful AI tutor, and simulated students can help researchers evaluate whether tutors can accurately identify and respond to those changes. Looking forward, researchers can further validate this approach by investigating whether tutors that perform well with high-quality simulated students also perform well with real students.
Understanding changes in a student’s skills and knowledge is a key component of a successful AI tutor, and simulated students can help researchers evaluate whether tutors can accurately identify and respond to those changes.
Building Accountability and Trust
The learning-by-teaching systems that inspired simulated students were not designed to evaluate AI tutors. But they demonstrated that if AI agents can represent learners, they can help create a more rigorous standard for evaluating educational technology. This could provide clearer evidence about whether AI tutors are aligned with learning goals before adoption decisions are made. Rather than relying primarily on product demonstrations or limited classroom trials, schools and educators could have access to evaluations showing how AI tutors perform across different learning goals and student needs.
Simulation-driven AI chatbot development makes it possible to iterate more quickly and enter real-world testing with stronger, more refined AI tutors. This can ultimately give educators and school leaders greater confidence that the AI tutors they adopt are built around meaningful learning outcomes and have demonstrated potential as effective learning partners for students.
1 thought on “How Simulated Students Can Build Better AI Tutors”
Hi Julia
Great post regarding AI Tutors, Just shared it with my team at Think Study learn.
We are working on AI tutors as well. Happy to show you a demo if interested, in the hope that we can share knowledge and collaborate for better Quality AI Tutors which dont hallucinate and have minimal latency.
Hope this is of interest to you.