The Learning Engineering Virtual Institute, or LEVI, is a collaboration of researchers, learning engineers, and educators striving to develop, scale, and implement new tutoring platforms that can double the rate of math progress among middle school students, especially those from low-income backgrounds. Since 2022, seven teams have been working to achieve this goal. Their ideas range from an AI-powered chatbot that provides personalized math tutoring to AI video technology that digitally replicates the experience of having a personal tutoring session.
One team, at the University of Florida, is building the ALTER-Math learning platform, which will employ a “learning-by-teaching” approach. This means students won’t just learn math – they’ll act as teachers themselves, becoming active participants. ALTER-Math will use AI-powered “teachable agents,” powered by advanced language models, to help students learn. It will be part of Math Nation, focused on algebra, and could reach millions of students. In this installment of “5 Questions,” team leader Dr. Wanli Xing discusses their work and findings thus far.
What was your “ah-ha” moment when you knew you were on to something workable?
Before starting the ALTER-Math project, I had been working on retrieval-augmented generation (RAG), primarily as a tool to reduce LLM hallucinations – like most in the field. However, a lunch-break insight changed my perspective. I realized that building effective LLM-powered teachable agents relies on a unique form of “hallucination”: the ability to accurately reflect and integrate what their student teachers have taught them. With this realization, I saw an opportunity to make these agents both highly effective and at a low cost through the use of structured external memory. Leveraging my expertise in graph-based algorithms, I quickly developed a pipeline that uses LLMs to build knowledge graphs, which store and retrieve the memory of teachable agents. And it worked! Now this has been a major direction of our research and development.
Have you made any significant shifts or course corrections while working with LEVI?
We have not seen major shifts, and we remain confident in the promise of AI-powered teachable agents, as shown by our A/B tests and pilot studies. However, I have identified a new, more efficient approach for building these agents with LLMs. The challenge with our original method, which relied on reinforcement learning from human feedback (RLHF), was its high temporal and financial costs. While RLHF is effective for personalizing teachable agents based on student-AI interactions, our early prototypes revealed significant expenses and delays for finetuning models. In contrast, our new approach, using knowledge graphs, has demonstrated comparable results at a fraction of the cost and with minimal wait time, allowing real-time agent knowledge retrieval and updates.
What’s been the most surprising thing your team has learned thus far?
Our project is based on well-established learning science principles that have been shown to produce significant effects on student outcomes, particularly in STEM education. So, it was not surprising when our quasi-experiment with over 1,600 students showed that using ALTER-Math led to substantial learning gains – 1.56 times higher than typical improvement rates. What did surprise us, in a positive way, was that these benefits were consistent across students with varying levels of prior knowledge. Research suggests that lower-performing students can benefit less from educational technologies compared to high-performing peers. However, in the case of ALTER-Math, there was no such disparity in learning outcomes. Although this finding requires further validation through randomized and longitudinal studies, we believe it can be attributed to the inclusion of mentor agents and the use of LLMs in ALTER-Math. These features likely made the learning-by-teaching process more scaffolded, personalized, and engaging, especially for students who needed extra support.
When did you see your intervention working for students or teachers?
ALTER-Math was designed to meet the needs of middle school math students through well-documented learning theories and a co-design process with teachers from our target audience. Using a design-based research approach, we iterated on development based on continuous feedback from both students and teachers. Starting in late 2023, we conducted small-scale classroom studies focused on usability and motivation, and the results have been promising. Both validated scales and interviews showed that students and teachers liked the tool. Further reinforcing this, a quasi-experiment revealed that students using ALTER-Math experienced 1.56 times greater learning gains compared to the control group. This significant effect is encouraging and underscores ALTER-Math’s potential to support student learning effectively. However, this also opens up new avenues for exploration, particularly in optimizing the design of AI-powered teachable agents at a lower cost. For example, we’re investigating whether agent personality or the use of knowledge graphs can enhance the student learning experience further.
What do you anticipate your project accomplishing in five years?
Over the next five years, we anticipate four major deliverables:
- Robust evidence of the doubling effect of AI-powered learning by teaching, validated through high-stakes assessments and randomized controlled trials
- Broad project impact, reaching over 100,000 middle school students and their teachers
- An effective, low-cost, and easily customizable software development kit for teachable agents, allowing seamless integration into other math learning platforms
- A responsible, reusable, and scalable cyberinfrastructure—comprising LLMs, datasets, and codebases—that supports the educational community in advancing fair, accountable, and transparent AI for learning environments.
Kent Fischer
Communications Director