Introduction
On the heels of the COVID-19 pandemic, tutoring programs have emerged as a key strategy for addressing the significant learning losses experienced by students across the United States. With billions of dollars in federal funds committed to national educational recovery efforts, public schools are investing heavily in different methods to effectively implement tutoring at scale.
A recent study conducted by researchers from Brown University, Old Dominion University, and Vanderbilt University assessed one such school district’s efforts. In 2021, Metro Nashville Public Schools (MNPS) launched a district-operated tutoring initiative called “Accelerating Scholars.” Over five semesters, researchers monitored and evaluated the program’s implementation, scalability, and impact on student learning outcomes.
The initiative had a promising design and thoughtfully incorporated and evaluated tutoring innovations into the local school system. However, the results on student learning outcomes were mixed, and overall below what would be expected given other small-scale and large-scale efforts.
The initiative had a promising design and thoughtfully incorporated and evaluated tutoring innovations into the local school system. However, the results on student learning outcomes were mixed, and overall below what would be expected given other small-scale and large-scale efforts. The Accelerating Scholars program largely did not replicate the successes seen in smaller-scale tutoring studies. Researchers found that, averaged across all students, the tutoring program produced only minimal positive effects on reading and had no significant impact on students’ math scores. However, when disaggregated into student ability level and by subject, there were significant gains found, albeit at a small effect size.
These results should not be taken to indicate that tutoring innovations are ineffective; instead, the approach taken – in which grade-level material was provided to students, and results averaged across all populations, should be reconsidered in future approaches in favor of targeted approaches with a broader scope of tutoring materials.
Background
School districts across the country have partnered with third-party tutoring providers to use federal funds directed at COVID recovery services. Programs using this funding, according to the researchers, have been implemented in two primary models:
- On-demand tutoring, available outside school hours via text, phone, or web platforms but with low participation; and
- A more structured approach, integrated into the school day and aligned with the curriculum, which has generally had a higher impact on students.
However, researchers note that one of the main limitations with these approaches is the uncertainty of their long-term sustainability. Because they are largely reliant on federal funds, they are at risk of being eliminated when budget constraints arise. By operating its own program rather than relying on external parties, MNPS hoped to create a lasting and sustainable alternative.
District leadership hoped the program would be capable of meeting the varied academic needs of its highly diverse student population. Roughly 75% of the student body identify as students of color, with 38% identifying as Black and 34% identifying as Hispanic. Additionally, 40% of students are considered economically disadvantaged, and 20% receive English language learner services. Students with disabilities make up 14% of the district’s enrollment.
Program Design
The Accelerating Scholars program was designed to deliver consistent, high-dosage tutoring and integrate into the district’s core operations to improve student learning outcomes. The district hoped the initiative’s integration would help ensure its sustainability and promote alignment of its content with classroom instruction.
In the first semester, MNPS launched a small pilot, serving just 132 students across three schools. Over the following two years, the program expanded rapidly, delivering over 125,000 hours of tutoring to more than 6,800 students.
Researchers outlined several key design features of the program in the final program design:
- High-dosage: Students received a total of 90 minutes of tutoring per week in either two 45-minute sessions or three 30-minute sessions.
- Sustained duration: Students received tutoring for a total of 8 weeks in the first pilot semester, 10 weeks in the following 2 semesters, and 12 weeks in each of the final 2 semesters.
- Relationships: Students were paired with the same tutor for the semester to help create and foster strong relationships between them and the instructor.
- Curriculum alignment: Tutors followed a curriculum developed by the district, structured, standards-aligned ensuring that tutoring sessions reinforced classroom learning.
- School-based delivery: All tutoring took place in MNPS school buildings, primarily during personalized learning time or before/after school.
- Tutor training and support: All tutors completed a series of modules focused on program goals and academic content.
MNPS initially relied on volunteer college students and community members to serve as tutors in the program. Over time, however, the district began shifting to employing its own teachers, paraprofessionals, and other staff. By the program’s fifth semester, 85% of students receiving tutoring services received them from MNPS staff.
This approach played an important role in the program’s ability to scale and reach more students. The district found that its teachers and paraprofessionals were more familiar with the curriculum and better able to manage small groups of students compared to volunteer tutors. District staff were also able to collaborate with classroom instructors more easily during the day, which helped ensure alignment and consistency between the tutoring sessions and regular classwork.
District staff were able to collaborate with classroom instructors more easily during the day, which helped ensure alignment and consistency between the tutoring sessions and regular classwork.
Despite these benefits, there were also challenges when it came to staffing. For example, the reliance on district staff posed challenges in terms of recruitment and retention. While district employees received additional compensation for participating in the program, teacher burnout became a concern. Adding to their already heavy workloads, many educators were asked to use their planning periods or after-school time to tutor students. Scheduling conflicts also arose for students who wished to be part of the program but also received specialized services, such as special education between the tutoring program and specialized services, such as English learner services. This ended up limiting the number of students who were able to participate in the program.
The program also only offered support for topics in current lessons. While this approach is reasonable, it prevents these services from offering support to students who did have the requisite knowledge to succeed in these materials, and conversely students above grade level did not receive resources that would allow them to learn beyond what was in the classroom. This approach of teaching materials outside of current content is often a key feature in tutoring, in contrast to additional classroom time or practice.
Program Results
The study found that the Accelerating Scholars program led to only a small to moderate positive impact on average students’ reading scores – improvements ranged from 0.04 to 0.09 standard deviations. However, there were no significant improvements in math test scores for middle and high school students, nor any substantial impact on teacher-assigned grades in reading and math classes. Researchers’ heterogeneity analysis showed that, while the program had only minimal positive effects for students at the highest or lowest achievement levels, it had moderate to large positive effects for students in the middle of the achievement range – which are the students who were at the appropriate knowledge level for the materials covered in tutoring sessions.
The study found that the Accelerating Scholars program led to only a small to moderate positive impact on average students' reading scores and no significant improvements in math test scores.
Researchers outlined four key barriers that may have hindered the ultimate success of the program.
- Limited treatment-control contrast: Students in the control group worked with computer adaptive learning programs that provide personalized instruction, which may have offered significant benefit to students.
- Modest program duration: Many students received tutoring for only a semester rather than a full academic year, which may have resulted in fewer significant academic gains.
- Heterogeneous effects: Students in the middle achievement levels benefited most from the program, while those at the highest and lowest achievement levels benefited less.
- Miscalibrated expectations: Parties invested in the program may have overestimated the potential impact of tutoring at scale
Conclusion
Metro Nashville Public Schools’ Accelerating Scholars program became one of the first large-scale, district-operated tutoring initiatives to attempt to address pandemic learning losses. While hopes were high for widespread positive effects on student learning outcomes, the program largely did not meet expectations to replicate the large gains found in prior experimental studies. Still, the novel district-operated approach did lead to gains among certain student populations and outlined many additional promising practices that could be implemented in future tutoring programs. Both the positive and negative outcomes of the program can be taken as lessons for other districts around the country looking to design, implement, and scale tutoring programs that meet the diverse needs of students.
