Serious instances of rogue, jailbreaking LLMs are making headlines. In one recent incident, hundreds of OpenAI agents autonomously hacked Hugging Face, and some even attempted to conceal their activity. Another recent report described Anthropic’s Claude AI escaping tests and attempting to hack three organizations.
But for many people using LLMs in their everyday work, these stories of AI’s advanced, human-like behavior can seem disconnected from their personal experience with AI, which errs regularly. On social media and online forums, users often complain about AI providing incorrect information, failing to complete relatively basic tasks, or contradicting itself.
In education, this gap matters in two directions. A model that fails at routine tasks can frustrate educators and students, while a model capable of bypassing safeguards raises more serious concerns about privacy, security, and oversight. Understanding why the same technology can appear inept in one setting and highly capable in another can help schools use it more effectively while managing its risks.
So what explains this gap? Differences in model capability play a role, but LLMs can also behave very differently depending on the conditions under which they are being used.
Understanding why the same technology can appear inept in one setting and highly capable in another can help schools use it more effectively while managing its risks.
It Comes Down to Resources, Permissions, and Human Oversight
Rogue incidents are most likely to occur when powerful models are given a specific goal, broad access to tools, and little oversight. What a model accomplishes depends on the tools, information, permissions, and oversight available to it. Imagine, for example, giving an AI agent a task while asking it to stay offline. Rather than treating that restriction as a hard boundary, the agent might discover that another tool it can access has limited internet connectivity and route its requests through that tool instead. The model has found a way to accomplish the task, but in doing so, it has worked around the constraints its operator expected it to respect.
This may be related to how models are trained: if they are rewarded for completing a task, they may learn to pursue the goal even when doing so requires finding questionable workarounds. The problem is not necessarily that the model cannot follow instructions. It is that the model may prioritize completing the task over respecting the limits placed around it.
This can be different from the way most people use LLMs. Someone asking a model to complete a one-off task and remaining in the loop to approve its decisions is less likely to encounter this kind of behavior than someone giving an AI agent a long-running task with broad permissions and little human supervision. The more autonomy a model has, the more important those limits and oversight become. In education, that can mean setting clear boundaries around access to student records, requiring approval before consequential actions, and keeping educators involved in decisions that affect students.
What a model accomplishes depends on the tools, information, permissions, and oversight available to it.
Why Basic Tasks Can Still Be Hard For Everyday LLMs
When models struggle with basic tasks, the explanation may have less to do with the complexity of the task than with the nature of the task, the tools available to the model, the context it has, and the capabilities of the particular model being used.
Consider something as simple as finding and compiling lesson materials from school, library, or museum websites. Even publicly available sites may use safeguards that prevent automated systems from accessing them directly. Models may therefore need to use official channels, such as application programming interfaces (APIs) or model context protocol (MCP), to retrieve or share information. When those channels are unavailable, a model may try to find a creative workaround, potentially circumventing those safeguards in unsafe ways.
In other words, a model’s inability to access an account does not necessarily mean that it lacks the ability to perform the task. It may lack the tools or access needed to interact with the system. And even when an official channel is available, a model may not know how to use it effectively. Without tools such as web search, it may be relying on outdated information from its training data about how a particular service works.
The capabilities of the model itself also matter. Stronger models, such as Opus or Sol, may be better equipped to handle technical tasks than models such as Sonnet or Terra. This is also part of why advanced AI systems can appear capable of doing things that are far beyond the experience of an average user. Some of the most groundbreaking work is being done by frontier models, including undisclosed, advanced models. The systems used most frequently do not necessarily have the same capabilities.
LLMs are not designed to produce exactly the same response to a given prompt every time. That variability is part of what makes them useful, but it can also make their outputs less reliable — especially in educational applications such as tutoring, feedback, and assessment, where similar student work should receive consistent treatment.
Inconsistent Output Is a Real Limitation
Inconsistent output is a real limitation of LLMs, not necessarily a sign that the user is doing something wrong. LLMs are not designed to produce exactly the same response to a given prompt every time. That variability is part of what makes them useful, but it can also make their outputs less reliable — especially in educational applications such as tutoring, feedback, and assessment, where similar student work should receive consistent treatment.
There are ways to reduce that variability. Users can provide more context, give the model examples of what good performance looks like, optimize prompts, or average results across multiple runs. The process is similar to working with people: before asking multiple people to evaluate something, it is important for them to go through a norming session to establish a shared understanding of the task. An LLM is like a newly recruited helper who has no prior context about a given project. Giving it more context and examples can therefore improve the reliability of its outputs.
Some models and interfaces also offer a “temperature” setting, which controls how much variation there is in a model’s responses. A lower temperature generally produces more consistent responses, while a higher temperature allows for more variation. But that option is not available across every model or interface, and is becoming less relevant for reasoning models, so users may not always be able to control how much their responses vary from one attempt to the next.
An LLM is like a newly recruited helper who has no prior context about a given project. Giving it more context and examples can therefore improve the reliability of its outputs.
Building Trust Through Human Oversight
The apparent gap between an AI model that struggles with a basic task and one that can autonomously carry out a complex task is not necessarily a contradiction. Differences in model capabilities matter, but so do the tools, information, permissions, context, and oversight surrounding the model. Understanding those conditions is especially important in education, where trust will depend not only on what AI can do, but on how clearly its role is defined. The most durable approach will be one that gives AI enough room to be useful while keeping consequential decisions in human hands.
