Automated Student Assessment Prize (ASAP) 2.0 represents a significant advancement in automated essay scoring evaluation, building upon the original Automated Student Assessment Prize competition from twelve years ago. This benchmark evaluates large language models on their ability to score student-written essays, a crucial capability for supporting educators and providing timely feedback to students.
Dataset Summary
The ASAP 2.0 corpus comprises source-based, persuasive essays from standardized writing tests collected from 6th, 8th, 9th, and 10th grade students across multiple states. Key characteristics include:
- Source-based essays: All essays required students to read and integrate source information into their responses. Essays are scored on a 6-point scale.
- Five distinct prompts: Covering various persuasive writing topics.
- Demographic diversity: Essays selected to maximize demographic information including English Language Learning (ELL) status, economic background, disability status, race/ethnicity, gender, and grade level.
- Quality scoring: Human-rated essays using standardized rubrics.
This benchmark evaluates how well models align with expert human raters who underwent norming processes to achieve reliable interrater agreement (IRR). Strong performance indicates a model’s output decision is more aligned with human rater judgments for this specific essay scoring task.
Evaluation Methodology
Models are evaluated using Quadratic Weighted Kappa (QWK), which measures agreement between predicted and actual essay scores. QWK ranges from 0 (random agreement) to 1 (complete agreement), with negative values possible when agreement is worse than chance. Unlike simple accuracy, QWK penalizes larger disagreements more heavily (rating a “1” as “5” is worse than rating it as “2”).
Our evaluation process includes:
- Subsampling: 500 essays selected via stratified sampling to maintain original score distributions
- Multiple runs: Each model evaluated across 3 runs to assess reliability
- Statistical ranking: Models grouped by performance tiers using confidence interval overlaps rather than point estimates alone
A basic zero-shot prompt was used to evaluate models’ performance. This approach was chosen to:
- Limit bias from example selection
- Replicate real-world scenarios where educators use models without extensive setup
- Establish baseline performance that practitioners can expect “out of the box”
The prompt provided models with four inputs: the assignment instructions, source text, scoring rubric, and student essay. Models were instructed to provide only a single numeric score (1-6) as the final output without any additional explanation.
Temperature was set to 0 for non-thinking models when available. Temperature controls randomness in model outputs – higher values (e.g., 0.7-1.0) produce more varied responses, while 0 produces more consistent outputs. Thinking models (marked with [T] suffix) do not have adjustable temperature parameters.
Performance Thresholds
There is no universally accepted threshold for QWK. It is important to consider a number of factors such as the distribution of data, the range of the scale, as well as the complexity of the task when deciding what constitutes an acceptable threshold.
However, the range of .7 is often used as the target threshold in many educational settings, including automated essay scoring. Additionally, Landis and Koch’s categorization of QWK ranges are also often used to gauge the level of agreement. Based on these two established standards, the performance of models are color-coded to suggest a general-level performance guide for the users:
- Strong Performance (≥0.70, Teal): Meets standards for automated essay scoring
- Good Performance (0.60-0.69, Green): Acceptable for non-high-stakes contexts
- Moderate Performance (0.40-0.59, Orange): Use with caution; likely requires human oversight
- Poor Performance (<0.40, Red): Not suitable for this task
Note: The 0.70 threshold applies specifically to operational systems where automated scores supplement or replace human evaluation. Reliable human interrater agreement is a prerequisite before deploying any automated scoring system.
Performance Overview
- Gemini 2.0 Flash offers the best balance – strong performance (QWK = 0.562), fast (0.73 seconds), and affordable ($0.25 per 1k essays).
- Thinking models may not be a good fit for essay evaluation – they can be significantly more expensive with minimal gains. Extended reasoning may second-guess accurate assessments or drift from rubrics.
- All models fall short of the 0.70 operational threshold, suggesting zero-shot prompting alone isn’t sufficient for high-stakes use.
- Google’s Gemini model variants are the top performers
- Gemini 2.5 Pro [T] achieves highest QWK (0.585), approaching the threshold for acceptable agreement in non-high-stakes environments (0.6)
- However, the increase in cost ($21.03 per 1k) and latency (19.5 seconds) needs to be considered
- Gemini 2.0 Flash shows strongest balanced performance: cost-efficient ($0.25 per 1k), fast (0.73 seconds), and performant (QWK = 0.562)
- Gemini models consistently show strong latency performance
- Thinking models do not show better performance across the board
- Claude Sonnet 4 [T] costs double ($15.43 vs $7.59) but performs similarly to non-thinking version
- GPT-5 variants (standard, nano, mini) show weaker performance compared to GPT-4.1
- This implies thinking overhead may hurt more than it helps for essay scoring
- Extended reasoning might second-guess initial accurate assessments
- More tokens may cause drift from rubric or overcomplicate holistic judgments
- Thinking models do not justify their higher cost and latency for this task
- Claude Sonnet 3.7 outperforms newer Claude Sonnet 4 (Rank 1 vs Rank 4), suggesting newer isn’t always better for this task
- Llama 3.2 3B Instruct was included to represent low parameter models (3b)
- These models typically output only the same scores across essays. Llama 3.2 3B, for example, only outputs scores 4 and 5
A Closer Look at Model Performance
To illustrate the differences between model tiers, we examined error patterns from three models from different performance groups: Gemini 2.5 Pro [T] (top-tier), DeepSeek Chat v3.1 (mid-tier), and Llama 3 8B Instruct (bottom-tier).
- Error Patterns
- Off-by-one errors are consistent across all models (~46-47%). This uniformity suggests the task itself contains inherent ambiguity – even human raters sometimes disagree by one point. The key differentiator is exact match accuracy, which drops from 47% for top models to 36% for bottom-tier models.
- Severe errors (off by 2+ points) separate model tiers most clearly. Lower-tier models make nearly 3× more severe scoring mistakes than top performers (18% vs 6%).
- Prediction Calibration
- Lower-tier models collapse toward certain scores rather than matching the true distribution. For example, DeepSeek assigns Score 2 to 60% of essays when only 32% actually deserve it, while Llama over-assigns Score 4 (39% vs 19% actual).
- Top-tier models maintain better calibration, with predictions closely matching the actual score distribution across all score levels.
Cost and Access
- All costs are based on the Open Router API pricing (a unified platform for accessing multiple AI models). Costs are calculated using the average number of input and output tokens used across inferences.
- Reasoning models are significantly more expensive due to reasoning token costs: essentially paying models to “think aloud” to themselves, allowing them to engage in deeper thinking processes and catch their own mistakes.
- Several open-weight models can be self-hosted for free if you have appropriate GPU infrastructure. However, most performant models require GPUs stronger than consumer-grade, making free self-hosting impractical for most users.
- Unlike commercial models (GPT, Claude, Gemini) which offer free web interfaces, open-weight models might require access through paid services like Open Router unless self-hosted – a potential barrier to testing despite low per-essay costs.
Practical Implications
These results represent baseline zero-shot performance. For your specific scoring task, you can improve results through:
- Few-shot prompting: Providing 3-5 example essays with your ratings helps models learn your patterns
- Custom rubrics: Adding detailed criteria and score-level examples from your context
- Iterative refinement: Test on small samples, review errors, adjust prompts
For practical use, it may be best to start experimenting with Gemini 2.0 Flash, as it offers the best balance of performance (QWK = 0.562), speed (0.73 seconds), and cost ($0.25 per 1k essays scored). Including example evaluations in your prompt will likely improve alignment with your specific scoring context.
Dataset and Context
This benchmark was developed through a collaboration between Vanderbilt University and the Learning Agency Lab, with support from the Bill & Melinda Gates Foundation, Schmidt Futures, and Chan Zuckerberg Initiative. The original Kaggle competition concluded in July 2024. The dataset is accessible through a Kaggle page and a GitHub repository.