Eedi’s Misconception Annotation Project (MAP) evaluates large language models’ ability to identify and classify student misconceptions in mathematics. This task requires models to assess answer correctness and detect underlying conceptual errors—a crucial capability for providing targeted educational feedback.
Dataset Summary
The MAP corpus comprises student responses to Diagnostic Questions (DQs), a specialized multiple-choice format designed to reveal mathematical misconceptions. Key characteristics include:
- Diagnostic Questions format: Each question features one correct answer and three carefully designed distractors that reflect common misconceptions
- Written explanations: Students sometimes provided written justifications for their selected answers
- Dataset size: 52,463 student responses with associated explanations
- Multi-tiered classification: Responses require evaluation across three distinct dimensions
- Answer correctness: Whether the selected answer is correct (True/False)
- Misconception presence: Whether the explanation contains a misconception (Correct, Misconception, or Neither)
- Misconception identification: The specific misconception present, if applicable
This benchmark evaluates how well models can score student response and identify specific mathematical misconceptions that experts have catalogued and labeled.
Evaluation Methodology
Models are evaluated using Mean Average Precision at 3 (MAP@3). This metric gives models three chances to predict the correct misconception, with diminishing scores after each incorrect prediction. For example, if the correct misconception is B and the model predicts [B, A, C] in sequence, it scores 1.0. If it predicts [A, B, C], it scores 0.5. If it predicts [A, C, B], it scores 0.33. The high MAP@3 scores achieved by top Kaggle competitors (0.90+) means their models successfully predict the correct misconception on the first try for the vast majority of student responses. This metric aligns with the ongoing Kaggle competition framework.
The evaluation process includes:
- Subsampling: 1,000 data points selected via stratified sampling to maintain category distributions and ensure rare misconception coverage
- Multiple runs: Each model evaluated across 2 runs (reduced from 3 after consistency analysis showed minimal variation)
- Statistical ranking: Models grouped by performance tiers using confidence interval overlaps
A basic zero-shot prompt was used – no example questions or misconceptions provided to models. This approach was chosen to:
- Limit bias from example selection
- Replicate real-world scenarios where educators use models without extensive setup
- Establish baseline performance that practitioners can expect “out of the box”
The prompt provided models with four inputs: the full misconception code list, question text, student’s selected answer, and student’s written explanation. Models were instructed to output up to three predictions in a structured format (Category:Misconception pairs, space-separated, following the format used in the Kaggle competition) without any additional explanation.
Temperature was set to 0 for non-thinking models when available. Temperature controls randomness in model outputs – 0 produces more consistent outputs. Thinking models (marked with [T] suffix) do not have adjustable temperature parameters.
There is no universally accepted threshold for MAP@3. The framework below reflects zero-shot baseline performance based on evaluated data:
- Functional (≥0.70, Teal): May reduce educator workload meaningfully
- Marginal (0.55-0.69, Green): Useful for flagging potential issues
- Weak (0.35-0.54, Orange): Limited utility, requires significant refinement
- Poor (<0.35, Red): Not functional
Note: Top Kaggle competitors achieve 0.94+ MAP@3 with extensive optimization. These thresholds reflect zero-shot evaluation constraints.
Performance Overview
- Thinking models show clear advantage (top 3 are all reasoning models)
- GPT-5 Mini [T] offers best value among top performers
- All models significantly trail competition winners
- Reasoning models show clear advantage
- Top 3 models are all reasoning models
- Reasoning models show roughly 15-30% performance improvement over top non-reasoning models
- GPT-5 Mini [T] offers best value among top performers
- Ranks 2nd overall with MAP@3 of 0.622
- A cost-effective thinking model at $1.26 per 1k inferences
- Gemini models show consistent mid-tier non-reasoning strength
- Gemini 2.0 Flash and Gemini 2.5 Flash rank 5th and 10th without thinking overhead
- Maintain strong latency performance (0.7 and 0.76 seconds respectively)
- Gemini 2.5 Pro [T] performs similarly to cheaper Gemini variants despite significantly higher cost
- Claude shows task-dependent thinking benefits
- Claude Sonnet 4 [T] shows meaningful improvement over non-thinking version (rank 5 vs rank 13) – unlike in ASAP where thinking didn’t help
- However, 5x cost increase ($8.25 vs $1.62) and 5x latency increase (10.2s vs 2.1s) requires consideration
- Performance gap with competition
- All models perform significantly worse than current top Kaggle performers (0.90+ MAP@3 required for top 1,000)
- This gap reflects the difference between zero-shot evaluation and competition solutions that use fine-tuned models, extensive prompt engineering, and ensembling techniques.
A Closer Look at Model Performance
To understand what separates top-performing models from lower-tier ones, we examined error patterns from three models: GPT-5-mini (top-tier), Claude Sonnet 4 (mid-tier), and Phi 4 (bottom-tier).
- Misconception Detection Patterns
- Some misconception categories (Duplication, Wrong Fraction, etc.) are difficult to identify for all models.
- The largest performance gap appears in identifying correct answer with valid reasoning:
- GPT-5-mini: 6.0% miss rate
- Claude Sonnet 4: 40.7% miss rate
- Phi 4: 66.0% miss rate
- Claude shows advantages in edge cases, outperforming on specific misconceptions, but cannot overcome GPT’s dominance on the most common scenario (identifying correct answer with valid reasoning).
- Ranking Strategy
- Success depends heavily on the accuracy of the first guess. The second and third guess contribute less to high performance:
- GPT-5-mini: 52.2% First guess accuracy → 92.1% of total MAP@3 score
- Claude Sonnet 4: 35.9% First guess accuracy → 87.5% of total MAP@3 score
- Phi 4: 14.8% First guess accuracy → 69.5% of total MAP@3 score
Cost and Access
- All costs are based on the Open Router API pricing (a unified platform for accessing multiple AI models). Costs are calculated using the average number of input and output tokens used across inferences.
- Reasoning models are significantly more expensive due to reasoning token costs: essentially paying models to “think aloud” to themselves, allowing them to engage in deeper thinking processes and catch their own mistakes.
- Several open-weight models can be self-hosted for free if you have appropriate GPU infrastructure. However, most performant models require GPUs stronger than consumer-grade, making free self-hosting impractical for most users.
- Unlike commercial models (GPT, Claude, Gemini) which offer free web interfaces, open-weight models might require access through paid services like Open Router, which can be a barrier despite their low cost.
Practical Implications
These results represent baseline zero-shot performance across all misconception types with basic prompts. For your specific misconception detection task, you can improve results through:
- Restructuring task: For tasks like multiple-choice questions, answer correctness can be evaluated through rule-based logic rather than LLMs, allowing you to focus model usage on the more challenging misconception identification subtask, which may significantly improve model performance.
- Constraining the candidate space: The performance shown assumes evaluating all possible misconceptions. If you can narrow the candidates based on question-specific patterns, you may improve accuracy by reducing the search space to only the relevant misconceptions per question.
- Few-shot prompting: Providing examples of your common misconceptions with explanations helps models learn your taxonomy.
- Iterative refinement: Test on small samples, review misclassifications, adjust prompts and examples.
For practical use, GPT-5 Mini [T] offers the best balance of performance (MAP@3 = 0.622), reasonable cost ($1.26 per 1k), and acceptable latency (14 seconds).
Acknowledgements
This benchmark was developed with the support from the Gates Foundation and the Walton Family Foundation, and data from Eedi – an edtech platform that helps students ages 9-16 master math by identifying misconceptions through diagnostic questions, AI insights, and one-on-one tutoring.