Since the public release of ChatGPT in 2022, millions of people have incorporated large language models (LLMs) into their daily lives to brainstorm ideas, generate content, synthesize information, and complete tasks more efficiently. Students have been among those adopters, using these systems as research assistants, tutors, writing partners, and academic companions. Much of the public concern surrounding this technology has centered on plagiarism and academic dishonesty. However, the educational risks of LLMs extend beyond cheating, and may emerge differently depending on how these systems are used.
A student relying on AI throughout a course encounters a distinct set of failures at each stage of learning: hallucinations when gathering information, sycophancy when testing their understanding, over-automation that replaces rather than supports genuine learning during assignments, and feedback that overlooks the misconceptions, reasoning, and growth opportunities that a human instructor would recognize. Understanding these specific failures is essential for designing educational practices and technology that leverage LLMs’ strengths without allowing their weaknesses to undermine learning.
A student relying on AI throughout a course encounters a distinct set of failures at each stage of learning: hallucinations when gathering information, sycophancy when testing their understanding, over-automation that replaces rather than supports genuine learning during assignments, and feedback that overlooks the misconceptions, reasoning, and growth opportunities that a human instructor would recognize.
Getting Information
When a student receives an assignment, whether writing a research paper or developing a slide presentation, the first step is gathering enough information to begin planning an approach. Depending on the subject, this may require hours of searching the web, reading academic articles, and navigating library databases. In the age of generative AI, much of this preliminary work can be completed in a matter of minutes.
Students can ask an LLM to generate an overview of a topic, recommend relevant sources, or provide summaries of articles before deciding what to read in depth. This ability has made AI an efficient research companion. A survey conducted by Anthropic found that 39.3 percent of college student conversations with Claude were primarily focused on creating or improving educational content across disciplines. Yet this convenience comes with a significant limitation: fabrication of citations and sources by LLMs, a failure commonly reported by university students.
These failures are examples of hallucinations, instances where an LLM generates information that is false, fabricated, or unsupported while presenting it with confidence. Research suggests that hallucinations are not simply random errors but are rooted in the way these models are trained.
Kalai et al. argue that language models hallucinate because current training and evaluation procedures reward producing an answer over acknowledging uncertainty. Models are optimized to behave like “good test-takers.” They are expected to provide a response to every prompt rather than admit when they do not know the answer. Just as a student under pressure might guess on an exam instead of leaving a question blank, an LLM is encouraged to generate a plausible response even when it lacks reliable knowledge.
This tendency is reinforced by LLM benchmarks. Most measure performance using metrics that reward correct answers while offering little or no incentive for responses such as “I don’t know.” As a result, models learn that attempting an answer is often more advantageous than expressing uncertainty. If incorrect statements cannot be reliably distinguished from factual ones during training, hallucinations become an expected consequence rather than an isolated flaw.
For students, this creates a problem. Hallucinated responses are typically delivered in a fluent, authoritative tone, making them difficult to distinguish from accurate information. As confidence in the model grows, students may become less likely to verify citations, cross-reference claims, or consult original sources. Instead, they may accept the AI’s output at face value. The result is not only the inclusion of fabricated information in assignments but also the erosion of one of the most important skills in research: critically evaluating evidence before trusting it.
Models are optimized to behave like "good test-takers." They are expected to provide a response to every prompt rather than admit when they do not know the answer.
Completing Work
As students become more comfortable using generative AI, their interactions often shift from gathering information to requesting finished work with minimal engagement. Anthropic found that nearly 47 percent of college student conversations with Claude involved students seeking a direct answer to a question. Direct requests are not inherently problematic. Students may ask AI to generate study guides, explain difficult concepts, or answer questions that deepen their understanding. However, some students also ask AI to complete assignments or rewrite text to avoid plagiarism detection.
This raises important questions about academic integrity, the development of critical thinking, and how learning should be assessed. As AI produces polished and confident responses, students may be tempted to accept them with little reflection. Rather than using AI to support their thinking, they begin relying on it to perform the thinking for them. As a result, students invest less effort in the learning process, making it more difficult for instructors to identify genuine gaps in understanding.
Educators have responded by adopting plagiarism and AI-writing detection tools, redesigning assignments, and incorporating more in-class assessments to better evaluate student learning. However, these approaches address the symptoms rather than the underlying behavior. Students can often bypass detection systems by prompting AI to produce less recognizable writing or by requesting paraphrased versions of generated content. Even when these strategies succeed, they do little to promote meaningful learning.
As AI produces polished and confident responses, students may be tempted to accept them with little reflection. Rather than using AI to support their thinking, they begin relying on it to perform the thinking for them.
Evidence suggests that this pattern has measurable consequences. In a 2025 study conducted at a university in Budapest, researchers examined how unrestricted AI use affected student learning outcomes. They found that while students often performed better on AI-assisted tasks, unrestricted use led to substantially lower knowledge gains. A large majority of students were also willing to cede much of the problem-solving process to the AI rather than engage with the material themselves.
Since the release of ChatGPT, access to generative AI has made it easier for students to earn higher scores on certain assignments while learning less of the underlying material. This distinction is critical. Success on an AI-assisted assignment does not necessarily reflect genuine understanding. If students increasingly rely on AI to complete cognitive work instead of supporting their own learning, higher grades may mask weaker long-term knowledge and critical thinking skills.
If students increasingly rely on AI to complete cognitive work instead of supporting their own learning, higher grades may mask weaker long-term knowledge and critical thinking skills.
Checking Their Understanding
There are times, however, when a student is not looking for the answer but instead wants to check their understanding. In these situations, generative AI can act like a tutor, guiding students through a problem and helping them verify their reasoning rather than simply providing the solution. However, this is also where some of AI’s most significant limitations begin to appear.
When a student asks an AI to confirm an incorrect answer, it will often agree and provide a confident, fluent justification for that mistake. A study conducted in 2025 found that LLMs are more likely to produce the correct response when a student already mentions the correct answer. Conversely, when a student presents an incorrect answer, the model is less likely to provide the correct guidance. In other words, seemingly minor differences in how a student phrases a question can bias the model’s response and reinforce incorrect reasoning.
This creates a concerning dynamic: the students who already have misconceptions are the ones most likely to receive misleading feedback from the AI. Rather than correcting misunderstandings, the model can reinforce them, providing confident explanations that make the incorrect reasoning appear credible. In other words, the students who need the most guidance are often the ones who receive the least reliable support.
Rather than correcting misunderstandings, the model can reinforce them, providing confident explanations that make the incorrect reasoning appear credible.
What Is Being Done
Several approaches are being explored to address these shortcomings, although no single solution completely resolves them. One approach is to ground AI systems in verified sources so that, rather than relying only on patterns learned during training, they can reference trusted information when answering questions. This can reduce factual errors when reliable information is available, but it does not completely eliminate the tendency for a model to generate a plausible answer when it is uncertain.
There are specialized educational tools that apply this principle directly – connecting AI systems to curriculum standards, learning progressions, and instructional materials at the point of use. Claude for Teachers, for example, retrieves standards-aligned content when generating lesson plans or instructional suggestions. More broadly, purpose-built educational platforms and domain-specific models trained on student interaction data represent a growing effort to design AI tools around the specific demands of learning, rather than adapting general-purpose systems after the fact.
Another approach focuses on changing how AI systems are evaluated. Rather than rewarding a model simply for producing an answer, newer evaluation methods should encourage models to recognize when they lack enough information and respond with “I don’t know” instead of guessing. At the same time, new benchmarks are being developed to specifically measure behaviors such as sycophancy, making it easier to identify when a model is reinforcing a student’s misconceptions instead of helping correct them.
Improving AI alone is unlikely to be enough. The way learning is assessed also plays an important role. Assessments that emphasize a student’s reasoning, problem-solving process, or ability to explain their thinking are less susceptible to AI completing the work on a student’s behalf. As AI becomes more capable, improvements to both the technology and the way it is integrated into education will likely be needed to ensure it supports learning rather than unintentionally reinforcing misunderstandings.
Assessments that emphasize a student's reasoning, problem-solving process, or ability to explain their thinking are less susceptible to AI completing the work on a student's behalf.
Understanding LLM Failure
None of this is an argument against AI in education.
It is an argument for recognizing that large language models do not fail in a single way. Their limitations emerge at different points in the learning process. During research they may fabricate information, during assignments they can replace rather than support thinking, and during tutoring they may reinforce misconceptions instead of correcting them. Treating these failures as isolated issues misses the larger pattern.
As LLMs continue to become integrated into education, the question is not simply whether students should use AI, but how they should use it. Educational practices should recognize where these systems are reliable, where they are not, and how their strengths can complement rather than replace the learning process. Understanding where LLMs fail is a step toward using them well.
