The organization: The Learning Agency Lab, the sister organization of The Learning Agency, was dedicated to developing science-of-learning-based tools and programs for social good. Its mission was to help educators, researchers, and others better understand how students learn, and how they can learn more effectively. The Lab team has since been integrated into The Learning Agency with the work continuing via our Research and Development team.
The need: The Feedback Prize was created to meet a pressing challenge: the need for advanced, ethical, and accurate automated writing assessment tools. Writing is a critical skill for student success, yet many students don’t write often enough, and when they do, they rarely receive the timely, specific feedback necessary for growth.
One key barrier is the time and resource constraints teachers face when grading writing assignments. These constraints are especially harmful for Black and Hispanic students, who are disproportionately affected and more likely to write at the “below basic” level compared to their white peers.
To tackle these challenges, the Feedback Prize leveraged data science and open competitions to spark innovation in assisted writing feedback tools (AWFTs), with the goal of helping students from all backgrounds strengthen their writing skills.
Our role: In collaboration with The Learning Agency Lab, Georgia State University, and Vanderbilt University, the Feedback Prize delivered on its mission to advance equitable, high-quality writing feedback through AI. Structured in two phases, the project combined field engagement with technical innovation, resulting in several successes:
-
-
Curated two open source datasets for research and product development: To fuel innovation in assisted writing feedback tools, the team developed two large-scale, annotated corpora of student writing, comprising approximately 32,000 essays curated from eight different organizations and states:
- PERSUADE, a dataset of over 25,000 argumentative essays from grades 6-12, labeled for key discourse elements like claims and evidence.
- ELLIPSE, a set of over 6,000 essays from English learners in grades 8-12, labeled for language proficiency traits like vocabulary and grammar.
-
Informed by teacher input, both datasets have been made available to researchers and developers on The Learning Exchange and Github, laying the foundation for future progress in AI-assisted feedback.
- Created high-quality models now integrated into educational tools: Winning algorithms achieved human-comparable accuracy in evaluating student writing. Notably, the top models from the first Feedback Prize competition reached an accuracy rate of 75%, comparable to the agreement rate between human annotators. These models have since been adopted into Google Gemini and other educational tools to enhance their performance.
- Teachers in the loop for meaningful AI development: Educators were central to every stage of the project. Through 70 interviews, a day-long conference with nearly 40 thought leaders, and 200 survey responses, their insights helped ensure the resulting algorithms and datasets were practical, relevant, and responsive to classroom realities.
- Drove innovation and community engagement through open competitions: Three open Kaggle competitions drew thousands of participants, pushing the boundaries of what AI can do in writing assessment. Collectively, participants invested an estimated $240 million worth of time, competing for a combined prize purse of $270,000. These competitions not only yielded high-performing models but also sparked sustained interest from the data science community.