Talk to people building educational tools with Automatic Speech Recognition (ASR) and you’ll hear two contradictory stories. Some will tell you ASR has never been better, that it’s finally ready to open up new ways of teaching and learning. Others will tell you it’s still not usable, that they’ve been frustrated by years of slow progress on the problems that matter to them. Who’s right?
Both of them.
This article focuses on one slice of the broader question: ASR for early literacy, specifically the tools used to teach young children to read. For some use cases, ASR is already good enough to build on. Many models can transcribe a child retelling a story, explaining their thinking about a text, or talking through what they noticed in a passage, even for young speakers and diverse accents. But try to use a popular large speech model like Whisper to capture a kindergartener sounding out individual phonemes, or to preserve the specific error a second-grader makes when decoding an unfamiliar word, and what you get back is usually gobbledygook.
So what’s going on?
For some use cases, ASR is already good enough to build on. Many models can transcribe a child retelling a story, explaining their thinking about a text, or talking through what they noticed in a passage, even for young speakers and diverse accents.
Two Kinds Of ASR Models
The familiar big models, Whisper and its relatives, are transcription models: give them audio and they return the most likely account of what a person said, drawn from everything they could have said. That’s an open-ended job. It’s not the only kind of speech model. Some early literacy tasks require something narrower: checking whether a speaker produced a particular, known sound. (For simplicity, we can call these identification models, though it is not a standard term).
A good transcription model is attractive, and the excitement about its potential for literacy education is understandable. You build it once, and then you can point it at almost anything, and it doesn’t need to know in advance what it’s going to hear. An identification model is the opposite. It’s built around a known target, so it does one job: it checks whether a particular thing was produced. Given the choice, it’s natural to reach for the general tool, and most of the time that instinct is sound.
The catch is that the same properties that make a transcription model so flexible are what make it unreliable for a particular class of tasks, including some of the ones that matter most in early literacy.
How Transcription Models Are Trained
Many transcription models like Whisper are trained on enormous collections of audio paired with text transcripts. We’re talking much of YouTube, audiobooks, podcasts, paired with transcripts that are sometimes written by humans and sometimes generated by earlier speech models. The task is straightforward: given an audio file, predict the transcript. The model is penalized every time its prediction doesn’t match the transcript that already exists.
The transcripts they train on are already cleaned up. False starts, “ums” and “ahs,” and repeated words have mostly been removed, so the transcript reads as tidier than the audio it came from. Real speech is far messier than a written transcript makes it look. Trained to reproduce the clean version, the model learns to do that tidying itself, skipping the disfluencies and returning smooth text.
The models also don’t work word by word. They predict whole sentences, so they may favor more likely, familiar phrases over an unusual or incomplete one. Getting there means picking up the statistical regularities of language, which words tend to follow which, and using the surrounding semantic and grammatical context to resolve sounds that are ambiguous on their own. You can watch this happen with any dictation tool: feed it a few isolated words and it stumbles, but give it a full sentence and the transcription sharpens, because now there’s context to lean on. The more context, the more the model leans on what should come next rather than on what it actually heard.
The result is a machine that takes in long, messy sound waves and returns clean strings of intended words. That’s enormously useful for most of what people want from speech recognition. But by training and by design, it’s built to discard the literal in favor of the intended.
Why That Isn't Always What You Want
For most of what people want from speech recognition, a transcription model’s habit of cleaning things up is a virtue. For assessing early reading, it’s disqualifying, because the mistakes are the data. A child reads “the cat sat on the mat” but says “sit” for “sat”; the model writes “sat,” since that’s the word the sentence calls for. The one piece of information you needed, that the child misread it, is gone. You don’t want the sentence the model thinks they meant. You want the “sit” they actually said.
How much of a problem this is depends on the task, because they differ in how much context they hand the model in the first place. The four activities below run from the most context to the least.
- Open-ended student speech: think-alouds, story retells, explaining reasoning. Plenty of context, and the teacher wants meaning, not literal words. Transcription models work well here, even for young speakers and diverse accents. The cleanup helps rather than hurts.
- Oral reading fluency: A child reads a passage aloud, a teacher tracks errors. Works moderately well, but the same context that helps the model also pushes it to “correct” reading errors back to the expected word. Doable, but it takes care to preserve the errors rather than smooth them over.
- Word-level reading: sight words, isolated words. Without surrounding words to lean on, the model loses the scaffolding it depends on, and accuracy drops. In principle this might be improvable; we’re less optimistic about sub-word.
- Sub-word-level reading (i.e., phonics): Did the child produce “th”? Can they segment c-a-t? Harder still, and a different kind of task altogether.
So why is sub-word analysis the hardest case?
For most of what people want from speech recognition, a transcription model's habit of cleaning things up is a virtue. For assessing early reading, it's disqualifying, because the mistakes are the data.
Phonemes Are Stranger Than They Look
Push down to the sub-word level and the problem is no longer the tension between what a student literally said and what they were trying to say. The task has changed entirely.
Analyzing sub-word speech from children is tricky for two separate reasons. The first has to do with the nature of a phoneme. Part of the task is learning the grapheme-to-phoneme mapping (a grapheme is a letter or letter-combination on the page; a phoneme is a distinct unit of sound), and that mapping isn’t clean. The same grapheme can map to more than one phoneme: “c” is /k/ in “cat” but /s/ in “city,” and “a” is a different vowel in “cat,” “car,” and “care.” Learning to read means learning these correspondences, and they don’t follow a simple rule. The second reason is that even for a fixed grapheme-to-phoneme relationship, the actual acoustic signal varies. The /s/ a five-year-old produces and the /s/ an adult produces are physically different sound waves, and subtle accents show up in the spectrograph as well.
For an individual who has already learned to read in the language they grew up with, resolving all this is instinctive; and you often don’t notice you’re doing it. For a model, it’s a genuinely hard problem, and not the one a transcription model was built to solve. None of it is unlearnable, but it’s the kind of fine-grained correspondence that an ASR model needs to be purposefully built to solve.
Why Fine-Tuning Existing Models Is Harder Than It Sounds
A logical reaction to the above points is to suggest that you just collect an hour or two of children performing reading tasks and use that to fine-tune one of the large transcription models. It’s a reasonable instinct, and roughly the strategy the field has leaned on, but it has two problems.
The first is that the data is genuinely hard to get. Most publicly available speech data is adult speech, and children’s voices differ in ways that matter, both acoustically (shorter vocal tracts, higher formants) and behaviorally (non-standard grammar, invented words, constant pauses). Collecting more is its own headache: classrooms are noisy, and voice is a biometric identifier you can’t strip of identifying detail the way you can with text. This problem is difficult but tractable.
The second problem is subtler, and it’s where the naive version of the plan falls down. In principle you could train a model to recognize phonemes; the target is well-defined enough. But the training data it would take is not a few hundred labelled clips you fine-tune Whisper on over a weekend. To teach a model the many-to-one mapping from a range of acoustic productions to a single phoneme label, you need a large, high-quality, carefully structured dataset, phonetically labelled rather than spelled out in ordinary text, and consistent enough that different expert labellers agree on what they’re hearing. That agreement is itself hard to reach. Assembling data like that is a serious, expensive undertaking, and skipping it, grabbing 150 examples and updating a general model, is exactly the shortcut that keeps not working.
There is a cost on the other side too. A model optimized for general transcription is not automatically suited to phoneme-level assessment. Adapting it may require changing what the model is trained to output or adding a separate step that compares the recording with the expected sounds. Improvements on this narrower task also may not translate into better general transcription.
How much fine-tuning it takes depends on the unit. For sentence- and passage-level work, fluency and connected reading, a modest effort might be enough: a few hundred examples, or some accent coverage, to nudge a transcription model toward holding onto errors instead of smoothing them, since the unit is big enough that there’s still something to transcribe. Word- and sub-word-level work is a different order of problem. There you likely need a large, serious dataset and a deeper kind of retraining, not a light fine-tune on top of a general model. Exactly where along that spectrum the line falls is an open question. What seems clear is that few-shot won’t get you there, and neither will waiting: this isn’t the kind of capability that emerges on its own as transcription models are scaled up and trained the usual way, any more than a model gets better at arithmetic just because you ask it to think out loud.
In a multi-round US data-science competition on word- and phoneme-level recognition, strong teams tried plenty of clever modeling, mixing models and the rest, but they were working from limited data, and they still landed nowhere near usable. The lesson isn’t that they lacked skill. It’s that backwards-fitting a big transcription model without a substantial, high-quality dataset behind it is unlikely to work, however good the engineering on top.
The Unsexy Infrastructure
High-quality open-source data. Good children’s speech data is what gates everything else, and it’s the one thing the general-purpose providers won’t build, since a niche this small won’t move their roadmap. The version worth funding isn’t raw volume but data built for identification: children producing known targets, letters, sounds, real and nonsense words, labelled and openly shared so any team can build on it instead of starting from zero. The Harvard Child-Centered AI Lab and its collaborators are beginning to assemble exactly this.
Better benchmarks. Most ASR benchmarks score on adult speech or general transcription, which tells you nothing about whether a model can handle a kindergartener reading one word aloud. A benchmark that splits identification from transcription and reports across the sub-word gradient, isolated letter-sounds, phonemes in connected speech, blending, would show plainly where progress is real and where it’s stuck.
Fast feasibility research. Traditional efficacy studies take months, long enough that a product usually locks in its design before the findings land. Faster, upstream research can test whether an AI tool can actually do a task before it’s built into a product, while pivoting is still cheap. EdTechnical’s research sprints are one version: short studies, run in weeks, that answer a feasibility question from existing student work rather than waiting on a full trial.
Catalytic funding. None of this comes from a single team, and for the first time the coordinating money is arriving. LEVI Literacy, led by Renaissance Philanthropy and The Learning Agency, aims to halve the number of struggling K-2 readers, and a $40 million investment from the U.S. National Science Foundation and Institute of Education Sciences is funding a multi-university consortium on early screening. If these funds are pointed at identification data and task-level benchmarks, rather than poured into ever-larger transcription models, they could build the infrastructure the field has been missing.
So What's ASR Good For? A Lot, In The Right Places.
The conditions for building good literacy tools are better than they’ve ever been. ASR already handles some literacy tasks well. The hard ones come with real obstacles, but they’re obstacles the field knows how to work on.
Most teams we see making real progress on child literacy aren’t taking a general transcription model off the shelf and fine-tuning it. Some build identification models that check a known target. Others train a bespoke model on phonetic transcripts. Either can be the right answer. What they share is that they’re built for a specific task on purpose-collected data.
The conditions for building good literacy tools are better than they've ever been. ASR already handles some literacy tasks well. The hard ones come with real obstacles, but they're obstacles the field knows how to work on.
So, in short, our recommendation is to stop treating this as one problem with one model, and start matching the model to the task:
- To transcribe a child’s think-aloud or story retell, the technology is already there. The remaining work is mostly privacy and recording quality.
- For low-stakes oral reading fluency, fine-tuning a mid-sized ASR model on 50 to 100 hours of children’s speech (some available in open-source corpora) gets you most of the way.
- Word-, letter-, and phoneme-level work needs purpose-built models and, behind them, purpose-built data. It won’t come from pointing a general transcription model at the task and hoping the next version handles it.
Each of these is a different problem, and the field will get further by treating them that way than by hunting for one model that solves all of them.
Ruth Schäfer
