AI-generated trials are not to be feared (opinion)

September 2022 was apparently the month when the artificial intelligence trial angst spilled over into academia, as various media published opinion pieces lamenting the rise of AI writing systems that will ruin the world. student writing and pave the way for unprecedented levels of academic misconduct. Then, on September 23, Academic Twitter exploded in a bit of a panic over this. The firestorm was sparked by a post on the OpenAI subreddit where user Urdadgirl69 claimed to get A’s with essays “written” using artificial intelligence. Professors on Reddit and Twitter expressed frustration and concern about how best to deal with the threat of AI trials. One of the most poignant and widely retweeted laments came from Redditor ahumanlikeyou, who wrote, “Noting something an AI wrote is an incredibly depressing waste of my life.

While all of this was happening online, my Rhetoric and Algorithms undergraduate students and I were running a little AI-generated student writing experiment. After reviewing 22 AI essays that I asked my students to create, I can tell you with certainty that AI-generated essays are nothing to fear. The technology just isn’t there, and I doubt it will be anytime soon. For the aforementioned AI essay activity, I borrowed an assignment sheet from the University of Texas at Austin’s freshman writing class. The assignment asks students to submit a 1,800-2,200 word proposal on a local issue. Students typically tackle issues on campus, coming up with ideas such as “It shouldn’t be so hard to get into computer science classes” or “Tuition should be lower” or “Housing on the campus should be more affordable”. For the purposes of the Rhetoric and Algorithms course, I asked students to rely as much as possible on AI. They were free to create multiple prompts to generate AI outputs. They were even encouraged to use these prompts in their essays. Students were also free to rearrange paragraphs, remove obvious repetitions, and clean up formatting. The main requirement was that they had to make sure most of the essay was “written” by AI.

The students in this class were mostly juniors and seniors, and many were rhetoric and writing majors. They did a great job, putting in a lot of effort. But, in the end, the essays they returned were not good. If I had believed these were real student essays, the best would have won somewhere around a C or C-minus. They met at least the posting requirements, but that was about it. Additionally, many of the essays presented clear red flags for the AI ​​generation: outdated facts about tuition, quotes from former college presidents portrayed as current presidents, fictional professors, and named student organizations that do not exist. Few students in my class have experience in computer programming. As a result, they mainly turned to freely available text generators such as EleutherAI’s GPT-J-6B. Several students have also opted to sign up for free trials of AI writing services such as Jasper AI. However, regardless of the language model used, the results were fairly consistently poor and generally quite obvious in their making.

At the same time, I asked my students to write short thoughts on the quality and difficulty of their AI essays. Almost all of the students said they hated this assignment. They quickly recognized that their AI-generated essays were substandard, and those who used to get top marks were loath to report their results. Students overwhelmingly said that using AI takes a lot more time than just writing their old-fashioned essays. To gain additional insight into the “writing” process, I also asked students to hand in any results collected from the AI ​​text generation “pre-writing”. Students regularly produced 5,000 to 10,000 words (sometimes up to 25,000 words) of results in order to cobble together essays that barely reached the floor of 1,800 words.

There has been a lot written about the supposed awesomeness of AI-generated text. There are even several articles, essays or even high-profile scientific papers or scripts written by AI that highlight this impression. In many of these cases, the “authors” have access to higher quality linguistic models than most students are currently able to use. But, more importantly, my experience with this assignment tells me that it takes a good writer to produce good algorithmic writing. Published examples are usually the beneficiary of professional writers and editors who craft prompts and edit the results in neat form. In contrast, many of my students’ AI-generated essays showed common problems with student writing — uncertainty about the appropriate writing style, problems with organization and transitions, and inconsistent paragraphs. Producing a quality essay with AI requires having sufficient command of the target writing style to create prompts that will cause the model to produce appropriate results. It also requires having strong organizational and editing skills. As a result, the best writers among my students produced the best AI essays, and the developing writers generated essays with many of the same issues that would have been in their authentic writing.

Overall, this exercise tells us that we are not about to receive a flood of algorithmically generated student submissions. It’s just too much work to cheat like that. The activity also tells me that the best defense against AI trials is the same as the best defense against trial deposits – a good homework sheet. If your assignment is “For today’s assignment, please describe the reasons for the American Civil War” (a literal stock prompt from the GPT-J template mentioned above), you are much more likely to get AI or downloaded essay submissions only if you write a detailed assignment sheet specific to the context of your class. The assignment I used for my Rhetoric and Algorithms students was quite a challenge because it asked them to address local issues of concern. There are simply not enough relevant examples in the data from which AI text generators draw to generate plausible essays on this topic as a whole.

Beyond academic misconduct concerns, this activity also showed me that using AI text generation can be part of good writing pedagogy. Two of the most important and hardest things to teach about writing are gender awareness and editing best practices. Developing writers don’t have the experience to understand the subtle differences between different types of essays or assignments. This is why student essays often seem overwritten or understated. Students are often still figuring out how to find the right place and how to adjust their style for different writing activities. Moreover, the usual delay between submission and feedback does little to help develop this intuition. However, rapid creation for AI text generators provides mostly immediate feedback. By experimenting with sentences that do and do not produce appropriate AI outputs, students can develop an idea of ​​how to write differently for different genres and contexts. Finally and unfortunately, most of my students complete their writing assignments in one session just before the deadline. It is difficult to get them to practice editing. The AI-generated text offers an interesting possibility for a sort of educational training exercise. Students might be asked to quickly generate a few thousand words and then turn those words into usable prose. It’s not “writing” in the same way that line drills aren’t basketball. But that doesn’t mean there isn’t a useful educational role here.

Ultimately, higher education will have to tackle AI text generation. Right now, most efforts to address these concerns seem to gravitate towards either AI evangelism or algorithmic desperation. I guess this fits more broadly with the AI ​​discourse. However, neither evangelism nor despair seems to me to be the ideal answer. For those despairing, I think it’s highly unlikely that we are (or soon will be) drowned in AI-generated trials. With today’s technology, it’s just too difficult and time-consuming than writing an essay. At the same time, I am deeply skeptical that even the best models will ever truly allow students to produce writing that far exceeds their current ability. Effective prompt generation and review depends on high-level writing skills. Even if artificial intelligence improves, I wonder to what extent novice writers will be able to steer text generators skillfully enough to produce impressive results. For the same reasons, I also question the enthusiasm of AI evangelists. It’s been just over five years since Google Brain computer scientist Geoffrey Hinton said, “We should stop training radiologists now. It is quite obvious that within five years, deep learning will outperform radiologists. Well, we’re still training radiologists, and there’s no indication that deep learning will replace human doctors anytime soon. In the same way, I strongly suspect that full robotic writing will always and forever be “just around the corner”.

Comments are closed.