Could a Vanderbilt University professor be on the precipice of derailing people’s ability to use ChatGPT as a plagiarism tool?
Linguist Scott Crossley, a Vanderbilt professor whose research focuses on using data science to explore language and cognition, is at the forefront of reinventing the science of writing by using keystroke data to better analyze the process of how people write. Collaborating with his graduate student, Yu (Terry) Tian, their work is poised to provide more insight into the specific writing characteristics that indicate originality, authenticity, and quality in essays and other writing samples. Their research also identifies common keystroke patterns used by effective writers, a potential key to understanding optimal writing strategies.
Yet their work may have more immediate implications for AI detection in writing as the debate continues among those who laud ChatGPT as a research tool and those who view it as a cheating engine.
Amid concerns over the new technology, one thing is becoming clear as education leaders weigh the implications of AI on learning: The use of ChatGPT as a vehicle for plagiarism may not go unchecked for long. A host of new research, ed tech platforms, and teaching techniques are emerging to prevent it.
A Path To Plagiarism Free
In the race to curb ChatGPT cheating, everyone from teachers to technologists are studying ways to ensure that original writing is not overtaken by bot-driven bunk in classrooms and workplaces. After all, many educators and philosophers consider the writing process, which requires synthesizing information, to be a cornerstone of critical thinking – one that people are wary of outsourcing to AI.
More emphasis on in-class writing is a path educators are increasingly exploring to limit ChatGPT plagiarism but online initiatives are also being utilized.
On the back-end of the plagiarism conundrum, more than a dozen AI detection tools exist that profess an ability to identify AI-generated materials. But in reality, these tools remain far from perfect and have become less perfect as generative AI evolves. A recent analysis by researchers across Europe and North America assessed 14 widely-used tools on their ability to accurately distinguish between AI and human-generated writing. None of the detection tools were accurate or reliable enough to use in practice, the researchers reported.
It is the opportunity to prevent plagiarism before students engage in it that is an emerging focus of those who study the science of writing.
What Keystrokes & Classwork Can Tell Us
Together with Crossley and Tian, our team at The Leaning Agency Lab recently ran a Kaggle competition to determine the current state of play in AI detection. We asked teams to assist in creating a tool that could distinguish essays written by middle and high school students from those generated by a large language model, such as ChatGPT. Up for grabs was $110,000 in prize money. The competition ended in January with more than 5,000 competitors – the 11th most participated-in competition on the Kaggle platform – a subsidiary of Google and a preeminent data science competition host.
Participants were encouraged to integrate the latest research and theories into their algorithms, with a goal of reliably determining whether the writing product (e.g., the essay itself) was written by a human or by AI. While the results produced strong models with accuracy around 99%, no model reached 100% reliability and many competitors expressed their surety that the detection markers they used, including unusual vocabulary use or strange phrasing, would be overcome within a few years by rapidly-evolving AI programs.
Enter keystroke logging and its potential applications. Keystroke logging focuses on the process or writing, not the product. Currently, keystroke research is in its infancy but shows promise. A Kaggle competition around keystroke data run by Crossley alongside the The Learning Agency Lab team, provided one of the first large-scale explorations into the relationship between learners’ keystroke logs and their writing performance.
Why could this be a gamechanger? Keystroke logging can provide a reliable method for detecting when students transcribe output from AI that is as accurate as current product based approaches but that demonstrates greater longevity because it focuses on the writing process, something AI cannot replicate. Thus, the evolution of generative AI will have no effect on the writing process, only the writing product Recent research by Crossley and colleagues demonstrates that authentic writing comprises a greater number of pauses before sentences and words, had a greater number of insertions and longer insertions, deleted more words and character, and had a greater number of revision than transcribed writing.
This process account of authentic writing supports what researchers have known about writing for a long time: writers monitor their writing and evaluate whether the writing needs to be changed if problems are identified. Transcribers, on the other hand, produced a greater number of writing bursts because they were simply copying language. Overall, the results indicated that authentic writing is a dynamic process while transcribed writing is much more linear.
Practical Applications
It may be months if not years until a truly effective AI-detection tool reaches classrooms. In the meantime, teachers are developing their own approaches, including capturing keystroke insights the old-fashioned way – by having students type in class (i.e., a wet ink approach).
“There has been an adjustment to writing instruction in the age of AI. For some teachers that means trying to focus more on in-class writing,” said Adam Sparks, a substitute English teacher in Nebraska who created a tool called Short Answer to support literacy and writing skills among middle schoolers.
Rather than having students write essays at home and risk the potential for AI plagiarism, in-class work offers a low-tech approach to monitoring keystrokes with additional benefits beyond verifying authenticity. Teachers can also support students as they write, answer questions and provide feedback. Student peers can do the same as a number of new writing tools are exploring.
In the end, whether a wet ink approach or using keystroke logs, the emphasis is the same – a push to have students write authentically. After all, when we write we think, we process, we synthesize, and we explain. And while ChatGPT can create efficiencies and may represent a virtual “easy button” for many mundane tasks, its ability to foster and spread AI-enabled plagiarism may create long-term learning pitfalls – pitfalls that researchers are now on a path to prevent.
This article first appeared on Forbes.com
