The Learning Engineering Virtual Institute, or LEVI, is a collaboration of researchers, learning engineers, and educators striving to develop, scale, and implement new tutoring platforms that can double the rate of math progress among middle school students, especially those from low-income backgrounds. Since 2022, seven teams have been working to achieve this goal. Their ideas range from an AI-powered chatbot that provides personalized math tutoring to AI video technology that digitally replicates the experience of having a personal tutoring session.
Working with schools in western Africa, a team from the Rising Academy Network has built Rori, a math chatbot that delivers individualized, high-quality tutoring through low-bandwidth, low-tech methods, like the popular messaging platform Whatsapp. The chatbot uses AI to gauge appropriate difficulty levels of math problems as well as natural language processing to deliver positive messaging and explanations about how math skills can be relevant to daily life or career aspirations.
In this installment of “5 Questions,” Hannah Horne-Robinson discusses the team’s work and findings.
What was your “ah-ha” moment when you knew you were onto something workable?
For me, it was working on our generative chat features. Whilst the math content is core to our product, the real defining aspect is the conversational nature of its delivery. We had brainstormed as a team ways to integrate some LLM-led conversations into the product but had so many questions and concerns about how to do this properly – both technologically and ethically. Working with LEVI really opened doors to answer these questions and pull on the expertise of so many people. Through workshopping our initial brainstorming ideas with experts at LEVI we were able to take these adaptive conversations from idea to reality and that was so exciting for Rori as a product.
Less from a product perspective, but it was a bit of an “ah-ha” moment when the JPAL project team visited a local government office and spoke to one of the managers for education in that area and he immediately agreed to our research plan. I am used to having to beg and badger policy officials for access to schools for even basic research, while this manager immediately requested staff to get the data we needed and took us himself to visit the schools. People are eager for solutions and have hope about Rori, so being in a context where people are actually likely to use our product is what makes it workable to me.
Have you made any significant shifts or course corrections?
We started by trying to give students lots of choices about what topics they covered. Since Rori is modeled after Teaching at the Right Level, we wanted to give students the choice to pick the topics that were most interesting to them. What we realized after looking at user behavior is that many students found this much choice overwhelming, weren’t always sure about where to start, and sometimes were practicing topics that were way too easy for them – because they liked getting quick hits of affirmation.
As a result, we are changing the design to be a bit more prescriptive about what level students should be working at, and are simplifying the pathways that students can move through the curriculum.
What’s been the most surprising thing your team has learned?
For me, the vast difference between school environments in Ghana and those that I have worked in previously means that I keep being surprised about how my assumptions do not hold up. Thus, all implementation planning has to be done carefully but with the flexibility to course correct, because I can not rely on any pre-formed ideas or educational experiences. This is not positive or negative, but rather about the little surprises that come up every day, like new problems to troubleshoot, exciting opportunities to explore, and unexpected student behavior.
When did you see Rori working for students and teachers?
Getting such a robust effect size from our Year One results was motivating, particularly at such an early stage. However, we learned that more careful consideration would be needed with this sample regarding both floor and ceiling effects. This has meant that I have to think very differently about crafting assessments than I have in other environments. For example, instead of trying to test students on the material we have covered, I want to ensure that every student can answer at least one question and have the opportunity to show all they know.
Also, a huge shoutout to our in-country teams because the insights and observations they share are crucial for us to be able to see the impact of Rori in schools across West Africa. One stand-out moment for me was watching videos of students in a real low-tech environment using Rori. These students weren’t absentmindedly answering questions on the phones just to move through the lesson, they were engaging and using pen and paper to do their workings out, and asking teachers and our implementation team questions when they were stuck. This was so eye-opening as it showed that students really were using Rori as a math resource and wanted to do well on the platform – ultimately it showed us the need to develop our hints and solutions so that Rori could further support students, even in instances when there isn’t enough teacher attention to go around.
What do you anticipate your project accomplishing in 5 years?
I really see Rori becoming an integral tool across schools in WA in the next five years in terms of supporting students in their maths learning to fill in any learning gaps.
Not only do I anticipate Rori will double math learning for middle school students in Rising schools across our network, but I hope that we can make a real impact in government schools in western Africa if we can figure out a few key technical and implementation requirements, such as having many students be able to use the same phone and ensuring Rori sessions fit well into the school schedule so that students can use it enough to really benefit.
Kent Fischer
Communications Director