Leiter Reports: A Philosophy Blog

News and views about philosophy, the academic profession, academic freedom, intellectual culture, and other topics. The world’s most popular philosophy blog, since 2003.

  1. Daniel Kaufman's avatar
  2. Samir Okasha's avatar

    Gabriele Taylor was my undergraduate philosophy tutor at St Anne’s College, Oxford from 1989-1992. I remember her as an incisive…

  3. Mark's avatar
  4. Santa Monica's avatar
  5. David Wallace's avatar
  6. David Wallace's avatar
  7. Daniel Kaufman's avatar

Moving from at-home writing assignments to in-class writing assignments to prevent AI cheating

Brian Leiter

MOVING TO FRONT FROM FRIDAY, DECEMBER 16–FINAL RESULTS, PRETTY REMARKABLE!

A philosophy friend on Facebook remarked that they were thinking of entirely replacing writing assignments that students would complete outside class with in-class writing assignments (essay exams, quizzes etc.).  Many others chimed in that they had already done exactly that.   Here's a poll for those of you who are teaching undergraduates:

If you teach undergraduates, have you switched from out-of-class writing assignments to in-class writing assignments because of concern about AI cheating?

 

 

 
 
pollcode.com free polls

Comments are open for those who want to explain their answers, give more nuanced answers etc.

UPDATE:  After four hours, here's what the votes look like:

f you teach undergraduates, have you switched from out-of-class writing assignments to in-class writing assignments because of concern about AI cheating?  
Selection   Votes 
Yes  45% 141 
No  19% 58 
Not yet, but thinking about it  36% 114 
313 total 
 

Comments, below, are also very interesting.  I have to say it's pretty dramatic that nearly half of respondents have already adjusted their evaluation modalities in response to Chat GPT etc.

ANOTHER (12/16/24):   Not much chage with nearly twice the votes recorded.  The results really are stunning:  more than 80% of undergraduate teachers have either already changed their assignments or are thinking about doing so because of on-line AI.  Maybe it's time to regulate the availability of AI, given how destructively disruptive it is to core educational functions?

If you teach undergraduates, have you switched from out-of-class writing assignments to in-class writing assignments because of concern about AI cheating?  
Selection   Votes 
Yes  47% 309 
No  17% 114 
Not yet, but thinking about it  36% 235 
658 total 
 

  

Leave a Reply

Your email address will not be published. Required fields are marked *

25 responses to “Moving from at-home writing assignments to in-class writing assignments to prevent AI cheating”

  1. Daniel Kaufman

    I’d already returned to all in-class assignments prior to Covid due to rampant cheating. Of course once Covid hit, everything went online, and I retired shortly after. The last few years were a dumpster fire of cheating and non-engagement.

    Reply
  2. Mark Taylor

    I have been influenced on this question by Lily King Abadal’s recent “Ensuring Genuine Assessment in Philosophy Education,” which I commend to others. Here's a passage which gave voice to my own guiding values when it comes to student writing (and why I answered "No" on the poll):

    “A term paper (albeit a properly scaffolded one) teaches students that thinking is a process. This ultimately gets to the root of what philosophy is—taking time to meditate on and attend to the “big questions.” These “big questions” are so properly named because they are of consequence to our human lives. As “big questions” of consequence, philosophical problems are not appropriate subjects for a thirty minute dash to produce a response paper right before the midnight deadline. Furthermore, they are not mere opportunities to demonstrate quick wit and intellectual prowess. Instead, they should be pondered, sometimes at an uncomfortably slow rate. The slow nature of engaging with big philosophical questions must be modeled for students through the structure of the assignments they complete. One important means to accomplishing this is by allowing students to pick one problem and think through it the entire semester—both in and outside the classroom. This demonstrates that something like the mind-body problem is bigger than one class period and merits months—at the very least—of questioning and deliberation before a reasonably good opinion has been developed on it.” (“Ensuring Genuine Assessment in Philosophy Education,” 259)

    Abadal ends up proposing assignments that are developed over many class periods with the instructor but are still completed at home. For my own part, I am convinced that, though philosophers may have to make such pragmatic concessions as a stop-gap, we must advocate in the long-term for university structures which allow a return to traditional essay development. Perhaps, like testing centers, schools need to set up writing centers where essay crafting must be proctored. I don’t know. But we need to make clear that a central skill of our discipline is undermined as long as students aren't taking the time work through these ideas slowly on their own. Radical and creative solutions will be needed, not resignation before inevitable technological "progress."

    Reply
  3. Peter Carruthers

    Students in my undergrad class write in-class exam essays as well as final-exam essays on their own laptops using the LockDown software that my university makes available within the online course Canvas system. (LockDown enables one to type, but nothing else — no copying and pasting, no accessing of any other internet sites, etc.) They can select their own topics from within a given range and can prepare their essays in as much detail as they like in advance. I let them bring in just a single index-card containing their notes which they can consult while typing up their paper. This seems to work pretty well. Typed essays as so much easier to read (with much greater fairness) than handwritten ones, and the quality of the work can be quite good. But length-of-essay is obviously restricted to what one can type up during the course or final-exam time.

    Reply
  4. DJL

    A colleague of mine has resorted to setting up an in-person, six-hour exercise where students are given a paper to read and then are asked to answer a number of questions about it, one of which requires composing a short essay – and all the while he is working on his computer. I'd like to do something like that soon.

    Reply
  5. David Silver

    I have been experimenting with requiring students to give in-class presentations of their works in progress. This forces them to treat writing and thinking as a process, allows them to observe and participate in the writing and thinking of their classmates, and gives me some confidence that their final written work represents their own efforts.

    Reply
  6. Rob Hughes

    This debate presupposes that an LLM can produce a competent essay. In the field I teach, business ethics, current LLMs do not give competent answers even to basic questions, such as, "Can you explain how Milton Friedman's shareholder theory is different from R. Edward Freeman's (2017) stakeholder theory?"

    I am still assigning traditional essays (in addition to in-class, paper-based multiple choice exams). For the final paper, I require in-class presentations which include a brief Q&A. My essay prompts require students to apply specific ethical theories to specific cases, so students are not responding to prompts that hundreds or thousands of students have answered before. Several students have gotten Ds or Fs in my courses because I caught them cheating with LLMs and had the evidence to prove it. There are other students I strongly suspect of using LLMs, but I gave them the benefit of the doubt. They did not get good grades.

    Reply
  7. James

    I switched to in-class exams for my 100-level courses when AI usage spiked while I figured out how to respond. (A surprising, possibly idiosyncratic observation: I was amazed at how much better the student writing was when done this way. Partly it was the absence of that bland ChatGPT voice, but even non-cheating students often wrote better this way. Possibly it has something to do with writing more slowly than with a keyboard or — increasingly — speech-to-text.) I didn't move away from at-home assignments for my upper-level courses because I had not noticed a particular increase in cheating there; occasionally a student would use AI in a way I could suspect, even if not prove, but overall I didn't find any reason to think that they were relying on it in the way or at the scale my 100-level students were overwhelmingly doing).

    This past semester I had three 100-level sections write four assignments at home, the prompt stating explicitly that AI usage was explicitly disallowed. The twist I added was that all writing assignments were related to one of the four movies we watched in class together and that the readings were then related to. While ChatGPT is quite good at bullshitting about written texts, it is (for the moment, at least) extremely bad and unreliable at describing in any detail scenes from movies. This was combined with rigorous citation requirements for the readings.

    The first batch of papers were maybe 8-10% obviously AI-written, with another set of papers maybe twice that size being questionable. The obvious cases were obvious not only because of that miserable ChatGPT voice that we all know, but because of the entirely fabricated scenes that ChatGPT provided (who can forget the scene in American Graffiti when Terry stands up to the former police officer in front of a crowd of fellow teens and passionately argues for his "right to enjoy his youth"? Or the wedding scene in Late Spring, a movie that is all about marriage but notably skips right over the wedding itself?). This kind of smoking gun evidence makes it much easier to talk to students about your concerns. For those cases that were obvious, I told students I'd need to meet with them to figure out how to grade it. About half of them pursued a meeting, and all of them admitted various kinds of academic dishonesty immediately (oddly, none of them admitted to AI usage as such; they all insisted they just used websites and didn't cite them, and none of them could ever seem to recall what website; they recognized that this was still academic dishonesty, but maybe they think it's less bad than AI usage? I'm not clear. Either way, the point might be moot, since so much website content is AI-generated that they might even be telling the truth.) For a first offense, I was clear about why this was both wrong and inadvisable prudentially, and consequences for the grade were worked out. The other half either stopped coming to class entirely or seemed grateful to accept a zero on the paper in lieu of an embarrassing meeting. I didn't have anyone who submitted an obviously AI-written assignment submit another obviously AI-written assignment over the course of the semester.

    For the batch of papers that were questionable (the scenes were either real ones from the movie — and ChatGPT isn't always wrong on that point — or the scenes were vague but not obviously phony), basically all of them failed to provide citations anywhere near the acceptable level. As I had stated on the prompt, I returned these ungraded and told them to enter citations and resubmit for late credit. The repeat offenders here are trickier; they learn to give the bare minimum of citations and attach them to the vagaries of ChatGPT's account of the philosophy readings. That said, when applying those philosophical ideas to the scenes from the movie, ChatGPT struggled since it normally didn't really know what scene was being described. Overall, the grades for these were never particularly good.

    I'm sure I missed some AI writing, but I'm sure I missed cheating pre-AI as well. What I found was that the incredible frustration and feeling of futility that was aroused when grading crappy AI paper after crappy AI paper when ChatGPT first hit the classroom was much more manageable when (a) I knew that the worst cases would be caught in an actionable way, (b) the less bad cases would get bad grades nearly all the time and would at least involve them having to bother to go through the readings looking for a way to provide citations, and (c) the majority of the papers displayed no evidence at all, at least to my eyes (and I spend a lot of time goofing around with ChatGPT), of AI usage. I suspect the increase in (hopefully) AI-less papers is at least partly due to the fact that I talked a lot about it in class, demonstrated a familiarity with how ChatGPT works, conveyed the disrespectful waste of my time and corruption of trust in the classroom that AI usage represented, shared stories of the dumb mistakes it makes and especially the fabricated scenes, warned them about the at least notional possibility of a watermark attached to AI writing, etc. I also had a number of students convey a real unhappiness that their peers were using AI and an understanding of the value of writing.

    Take from all that what you will. AI will get better at watching and describing movies and visual information. It will get better at providing accurate in-text citations. It will get better at applying theoretical claims to particular cases. Maybe this won't be an arms race one day, but it certainly is right now, and I was relieved to be able to spend my grading hours not wanting to strangle my students and then shoot myself (at least, no more than grading always makes you feel that way).

    Reply
  8. Don't be the change you don't want to see

    At this point it's hard for me to imagine the circumstances in which a philosophy instructor would be only "thinking about it" and not have already moved to in-class writing or other stop gap or long term measures.

    I am ashamed to say that I used chatgpt to help me make up a multiple choice exam this term, feeding it some of my handouts and other materials. Though some of the questions were good, most of the proposed answers were not great, and I went through and changed every question and all the proposed answers. Well almost all, I suppose. However, in retrospect I recognize that I did not entirely eliminate the voice, the chatgpt voice that some commentators have mentioned. A favorite student – a reflective person and an emerging writer, though entirely new to philosophy – caught me. He looked crestfallen sitting on a bench out in the hallway with friends after the exam. (He did fine on the test.) Later he sent me an email about returning a book that I am sure he did not write but which somehow contained a poignant message. I can hardly wait to talk to him and find out whether this is all my imagination or whether he detected it.

    Reply
  9. Don't be the change you don't want to see

    At this point it's hard for me to imagine the circumstances in which a philosophy instructor would be only "thinking about it" and not have already moved to in-class writing or other stop gap or long term measures.

    I am ashamed to say that I used chatgpt to help me make up a multiple choice exam this term, feeding it some of my handouts and other materials. Though some of the questions were good, most of the proposed answers were not great, and I went through and changed every question and all the proposed answers. Well almost all, I suppose. However, in retrospect I recognize that I did not entirely eliminate the voice, the chatgpt voice that some commentators have mentioned. A favorite student – a reflective person and an emerging writer, though entirely new to philosophy – caught me. He looked crestfallen sitting on a bench out in the hallway with friends after the exam. (He did fine on the test.) Later he sent me an email about returning a book that I am sure he did not write but which somehow contained a poignant message. I can hardly wait to talk to him and find out whether this is all my imagination or whether he detected it.

    Reply
  10. David Shoemaker

    One missing nuanced answer on the poll: "In part." I've made most of the lower-division written exams in-person (and those needing accommodations are proctored elsewhere with more time on the clock, typically). But because finals can be scheduled very close to Christmas, with lots of pleas to take one early so as to "make my plane back home," I still give them a "do at home" exam. I ran my pre-AI final exam prompts thru ChatGPT, and the bot spit back answers. Some were good, some were middling, and some were downright terrible or completely inaccurate. So my new final exam is to simply ask students to evaluate ChatGPT's answers to those old prompts (I flag relevant passages for them to evaluate). So far, so good. Recursion attempts with a request for ChatGPT (or other AI programs) to evaluate the first bot's answers have thus far failed to generate anything remotely close to viable.

    Reply
  11. Alex Miller

    I'm thinking of it, but the problem is that the "internal market" system whereby departmental budgets are set by student enrolments means that unless all departments do this simultaneously, the upshot could be that students gravitate towards subjects where they can work less by using AI. Another good reason why capitalism and higher education are not made for each other!

    Reply
  12. V. Alan White

    I'm retired, but since the 80s in my writing classes for philosophy I had my students do an in-class impromptu essay early in the course which I kept as a baseline for comparison for later out-of-class writing. It was helpful especially as I reminded them that I had that basis to see how they wrote. Plagiarism is nothing nothing new after all, just the tech improvements. I'm relieved to not have to deal with AI though. But I have to say I would move to all in-class writing with no other alternative. What pains me is that the mind-set of students is largely no longer to learn, but to post credentials of supposed learning by any means necessary.

    Reply
  13. Daniel Kaufman

    My students didn’t need LLMs to cheat en masse. They did it with Google, Wikipedia, Stanford, and other online resources. I’d returned to all in-class assignments before LLMs became the rage.

    Reply
  14. Menno Lievers

    Some of my own students who really love philosophy warned me and recommended the move back to in class writing assignments.

    Reply
  15. Hugo Viciana

    I’m doing both: more in-class writing and also assigning a scholarly essay for which they have the entire quarter to prepare and I supervise their partial progress. I believe relying solely on at-home writing or not supervising the progress in their preparation of at-home work would be a recipe for disaster with these young students (though I might be wrong—does anyone want to defend the opposite point of view?).
    On the other hand, focusing only on in-class writing also seems like a missed opportunity to teach them how to craft philosophical essays and develop philosophical work using modern technologies.

    Reply
  16. Gary Kemp

    Yes Alex! The situation is more or the same but on a much bigger scale in the UK. Competition between subjects/department for students but also between universities makes the situation all the tougher (calling for action at the government-level). In the US, by and large, the Professor has much more control; here — and I imagine in NZ but memories are hazy — every little piece of assessment has to be approved by the College (US: Faculty) boards, which partly take direction from the higher-ups.

    Reply
  17. R-1 Public Uni Prof

    I've had good luck so far using the techniques I've always used for essay prompts. In all my courses, my essay questions are *highly specific*, require cross-referencing within one reading and between readings, with footnotes everywhere the text is cited (so they have to cite our edition/translation. Yes, this requires teaching students how to format footnotes and cite text). Students submit a shorter first draft and then must submit a slightly longer final version which must respond in a relevant way to my comments on the draft. In courses of a manageable size (20-35), students must also hold an office hour with me (individual or small group office hours, in person or online) to discuss their developing ideas. Then they have to do a second essay on a different author starting midterm. So, they get about half a term to work up each of the two papers. Students also have to respond to weekly questions on the text on our website (around 300 words). The latter are not 'recall' questions either but are designed to lead them to make connections between themes, reconstruct an argument based on our text, assessing it for soundless, etc. (Something chatbots are very bad at doing!) Occasional very short in class quizzes are for 'recall'. I've only had a couple of cases of suspected cheaters using bots. (If I get more, I'm thinking of collecting these and using them in the following year's class, comparing them to well done essays. The difference is painfully obvious.) Students actually seem to appreciate that a professor is interested in their writing. Sadly, the only resistance I get is from senior faculty who have given up on teaching writing altogether, who express 'concern' that my courses are 'too' hard. (Perhaps they should retire?)

    Reply
  18. Steven Hales

    I'd like to see this broken down by type of college/university. I'm predicting the faculty at elite schools with top students are seeing less cheating and so are less worried about it, but those of us at lesser places are getting buried by cheating.

    Reply
  19. Craig Duncan

    I've always given midterm and final exams that require essays, and I am glad to continue this practice in this era of generative AI.

    However, I can't expect great insight and sophistication form a student in an essay that's generated in the fraught conditions of an exam, so I do still assign the usual 5-7 page, 6-8 page, etc. essays done at home.

    My response to AI so far has been the same response as R-1 Public Uni Prof above, namely, to make my essay prompts very specific, e.g. requiring textual quotations from specific articles that are the focus of the writing. That seems to help a bit, but it does not wholly eliminate AI usage. However, even this partial help will likely be short-lived. It's not long, surely, before AI will be able to be fed the relevant source materials and do a credible job with specific prompts like mine. (Maybe it is already possible for all I know…)

    So I see the appeal of writing essays in class, outside of an exam context, in order to truly prevent AI usage. But I worry that, as a learning exercise, these in-class essays are likely a pale shadow of the essays of old. A 6-8 page essay is 2000-2500 words, for instance. Are students really going to write that much in class? How many class periods will that take, and at what opportunity cost? Are these essays handwritten? Surely not, at least for long-ish essays, so does the instructor walk around as students compose their essays and police laptop usage in class to prevent surreptitious AI usage? I guess so.

    Relatedly, my own writing-style as an undergrad (though I'm aware this is atypical) was to think, write, rethink, rewrite, etc. — a laborious process that wouldn't have translated well to an in-class setting. I imagine there are other writing styles that would also translate poorly to an in-class setting.

    I suppose what I am asking is the following. For those of you who are using in-class writings: How precisely does this work? That is to say: Length (i.e. word count)? Modality (hand-written vs laptop)? Over how many class periods? What topics (more personal reflections or truly academic analysis)? Do these in-class essays replace traditional at-home essays, or merely supplement them?

    Reply
  20. Charles Bakker

    I am a fifth year PhD student who is starting to think seriously about course design as I prepare to spend the next several years in the job market collectors lane. As I see it, what we are trying to do is teach students how to think philosophically. What the administrations are trying to do is attract as many customers as possible. What governments are trying to do is implement ideological/political changes. What a majority of students are trying to do is get a good grade so as to get a good job. A small minority of students are not even interested in this. (Though it should be noted that this minority is growing in the age of social media influencers and skyrocketing costs of living.) And an even smaller handful of students actually want to learn philosophy. Without these latter students, why teach? Add to this mix the tech companies who are trying to maximize profits as they try also to manipulate the political landscape in their favour, and what you get is a bunch of different groups of people with different, often conflicting goals.

    I know this is going to ruffle some feathers here, but I think we need to go back to thinking of philosophy as a way of thinking about how to live life well, such that we can then explain to students how whatever (often arcane) topic we are covering, whether it be the history of medieval logic, or Kantian ethics, etc., connects back to their own ability to navigate life well. And more often than not, this will primarily involve explaining how the skills we are teaching them to develop are useful, even if they end up forgetting most, if not all of the content concerning philosophical topic X. (My pragmatism is probably showing here.)

    Bringing this back to the use of so-called “Artificial” Intelligence – as though natural intelligence was not itself social and instrumental, I intend to teach any students I might have how to use LLMs responsibly, effectively and ethically. For I am confident that this will prove to be a useful, if not even necessary skill.

    I am sure there was once resistance to using calculators in maths courses, on the grounds that students would forget how to really think mathematically. But this attitude would have failed to grasp the usefulness that calculators have for solving even more complex problems than would have been possible without their aid. And so it is with LLMs. These “sounds like an answer/paper” generators might help us to explore philosophical terrain hitherto unfathomable. I say “might” because it is still an empirical question as to whether this will turn out to be the case. Nevertheless, I think there are reasons which warrant our putting this possibility to the test. Do we not already pay attention to surveys of (expert) opinions? (PGR being one) Do the results of these surveys not already constrain our collective behaviour? As LLMs get better at tracking expert opinions, I see no reason why we ought not to try carefully using these tools to constrain our thinking, not just as philosophers, but as humans. Are there dangers lurking? Absolutely! But these dangers just speak to the need to learn how to use these technologies responsibly, effectively, and ethically. And that is surely better done in a philosophy classroom than on TikTok.

    I have no idea how to implement the sort of training I am describing, for I am literally one of the last philosophers to receive the bulk of my education in a world without easy access to LLMs. I guess that just means that I have some learning to do also.

    Reply
  21. Santa Monica

    Take a world and a college where

    (1) Most 1st year college students do not know what chess is (2) There are 100-level chess classes available (3) Such classes satisfy a core distribution requirement (4) Most of them involve 3-hour online classical chess games every few weeks (5) These games are graded on openings, tactics, blunders, etc. (6) Computer programs that can play chess well are widely available.

    Hypothesis: Chess cheating is a bigger headache for professors here than in nearby world-college pairs where (2) is false.

    If that is right and there is an analogy here, why not make philosophy a 200-and-up thing?

    Reply
  22. Don't be the change you don't want to see

    @ Santa Monica: I agree totally. Philosophy is wasted on those not ready for it. Making it a 200 and up thing would be a good idea. In fact, I'd go further: no one should be allowed to do philosophy until they're at least 25 and have had years of training in gymnastics, including all-gender mud wrestling; music, so they can learn harmony; mathematics, so they can practice discerning abstract patterns; plus a year in service to the public good. But after they've reached the age where they are allowed to do philosophy, philosophy should be a required course for everyone. In fact everyone should be required to take several such courses.

    @ Steven Hales: I also suspect that so-called "elite schools" with strong students are not feeling it, won't for awhile be feeling it, in the way everyone else is. Better students, more autonomy for the faculty, better culture with the presence of graduate programs. It would be nice to measure this somehow. The reason why this matters is that while R1s give us science, it's all the rest that give us democracy. By 'all the rest', I mean the regional universities, the community colleges, the night schools etc.. My assumption is that democracy is vulnerable when the general population is uneducated specifically in the values and practices one learns through studying philosophy. And the other assumption is that these values and practices can't be taught in an environment where AI is being rolled out as fast as it is being rolled out.

    Reply
  23. Don't be the change you don't want to see

    @ Santa Monica: I agree totally. Philosophy is wasted on those not ready for it. Making it a 200 and up thing would be a good idea. In fact, I'd go further: no one should be allowed to do philosophy until they're at least 25 and have had years of training in gymnastics, including all-gender mud wrestling; music, so they can learn harmony; mathematics, so they can practice discerning abstract patterns; plus a year in service to the public good. But after they've reached the age where they are allowed to do philosophy, philosophy should be a required course for everyone. In fact everyone should be required to take several such courses.

    @ Steven Hales: I also suspect that so-called "elite schools" with strong students are not feeling it, won't for awhile be feeling it, in the way everyone else is. Better students, more autonomy for the faculty, better culture with the presence of graduate programs. It would be nice to measure this somehow. The reason why this matters is that while R1s give us science, it's all the rest that give us democracy. By 'all the rest', I mean the regional universities, the community colleges, the night schools etc.. My assumption is that democracy is vulnerable when the general population is uneducated specifically in the values and practices one learns through studying philosophy. And the other assumption is that these values and practices can't be taught in an environment where AI is being rolled out as fast as it is being rolled out.

    Reply
  24. Daniel Kaufman

    I don’t know if this is a good comparison, Steven, but one reason I stopped teaching Honors classes is because the cheating was more common than in regular ones. I suspect it is because those students felt more of an imperative to get straight A’s. And I’ve heard anecdotally that cheating at the Ivies is rampant.

    Reply
  25. Santa Monica

    I wonder why philosophy is not taught in high schools.

    —–
    KEYWORDS:
    Primary Blog

    Reply

Designed with WordPress