Leiter Reports: A Philosophy Blog

News and views about philosophy, the academic profession, academic freedom, intellectual culture, and other topics. The world’s most popular philosophy blog, since 2003.

  1. Jason Stanley's avatar
  2. Daniel Greco's avatar
  3. Nobody's avatar
  4. Roger of Invisible America's avatar
  5. Santa Monica's avatar
  6. Optimistic about LLM's avatar
  7. Ben M-Y's avatar

More on LLMs from Jason Bridges: back to the blue book

My philosophy colleague Jason Bridges kindly gave permission to share his follow-up reflections about what his own approach will be in light of the power of the most recent version of ChatGPT and, in particular, why he is skeptical about “relying on relationships of trust to regulate AI use–essentially, using an honors system insofar as the classroom imposes any kind of policy limiting use.” Professor Bridges writes:

While I can recognize what is appealing about this approach, I am afraid I’ve become convinced that it isn’t workable in the long run. And because it is unworkable, I fear it is neither ethical nor in the best interests of our students or the College itself.

I thought I might offer my reasons for view, in case they may be of interest. I’ll note this topic has been especially in my mind as I am co-teaching a Philosophy of AI course this fall (we have 80 students signed up!), and as part of my preparation I have been exploring firsthand what current high-end LLMs can do.

The simple reason the ‘trust and honor’ approach is (I think) unworkable is the incredibly strong incentive structure pushing in the other direction–incentives that will only grow in force over the next few years. We are a year or two away from a time when even an ideal student–one who believes what we tell them about the importance of writing for their learning goals, who wants very much to learn, and has personal integrity–will have staring them in the face the following set of considerations:

1. An LLM can produce an A range paper with no effort on my part. [I’ve verified this with the recent ChatGPT5.]

2. It is easy to render the use of an LLM essentially undetectable.

3. Writing, even if worthy, is painful and difficult.

4. Grades matter enormously for my future.

5. Grades matter comparatively, and other students will be using LLMs.

Even if the counter-considerations appealing to their better nature and lofty ideals of learning suffice to fend off this line of thought through an assignment or two, the question will then be how things look after their first B grade. The pressure toward rationalization will become very, very strong.

There is a further issue embedded in all of this: namely, the need for institutions of higher learning to work to retain their credibility during a period where there are myriad forces arrayed against them.

I had occasion to talk with some students at prestigious high schools in the Bay Area this summer, and they reported that a large percentage of their classmates secretly use AI. They assume their teachers suspect this to be so even while the school seems unwilling to acknowledge this reality. This dynamic causes the school to lose respect in their eyes, and, I think, rightly so.

As I myself am convinced that the process of writing (for oneself, naturally) is crucial for learning how to think effectively in the fields I teach (and, by extension, for the kind of thinking that goes into effective citizenship), I cannot go for the alternative of allowing students to “write” with unbridled use of AI.

The upshot is that I’m moving to a blue-book based approach this fall–ironically, for my class on AI. I’m working out with my co-instructor the details now–we may experiment with supplementary take-home paper assignments that are not graded.

Another possibility for the College going forward is something like Exam4, which has long been used by law schools, allowing written exams to be done securely on a laptop.

I think he’s absolutely right that an “honor code” approach is doomed to failure. I myself am switching to a 4-hour in-class essay exam in my Jurisprudence class in the law school, after having used take-home essay exams for decades. We do use Exam4, or something like it, for in-class exams to block Interrnet access.

Leave a Reply

Your email address will not be published. Required fields are marked *

8 responses to “More on LLMs from Jason Bridges: back to the blue book”

  1. I agree on the switch to in-class exams. However, blocking internet access won’t be good enough if students are running an LLM locally.

    I don’t say this as criticism – I am genuinely wondering what the right approach is to prevent that type of cheating! (Just google “running llms locally” for a wide set of references on how to do it with many models, including models which are close to the frontier.)

  2. Junior faculty at a large public university

    I recently taught the ethics of AI, and I was astonished to find that at least half of all written work appeared to be generated entirely by AI. Strikingly, this was the case even for assignments graded on a pass/fail basis—which I had introduced precisely to discourage the kind of grade-fixated reasoning Bridges highlights—and even for students who were otherwise intelligent, interested, and motivated.

    This semester, I am using in-class blue book essays. I am also experimenting with a lockdown browser (in part to eliminate having to decipher unintelligible handwriting), but these seem to have a high rate of technical issues.

  3. I agree on the need to move back towards exams, but I’m a little less sceptical that essays are necessarily dead as a form of assessment. A university could, for example, require that students write their essays at designated desks in rooms with an invigilator where only physical books and articles are provided, writing the essay (perhaps over several different visits to the room) by hand or using university-provided computers with edit-tracking software and little or no access to the internet. Of course, this wouldn’t prevent a student from generating an essay with an LLM prior to going into the room, memorising it, and then pretending to write it… but it would be tricky to fake the standard writing style which edit-tracking software would be looking for. (And this would be not much different to the problem we faced with essay mills prior to the existence of LLMs.)

    What I think is a more worrying problem is that the companies generating, and pushing the use of, LLMs are going to create a world in which being able to write an essay (or basically any substantive body of text, of any type) will not be seen as a valuable skill people need to develop. I think that will be an unpleasant world in which to live.

  4. Switching to in-class exams is a poor response to the challenge of generative AI. I assign essays because I want students to take their time grappling with an idea or problem, engaging thoughtfully with existing scholarship on it, and crafting their own considered response. That intellectual work cannot happen under timed, high-pressure exam conditions, which typically produce rushed, formulaic responses that merely summarize course material.

    The widespread retreat to in-class exams is troubling trend that simply reinforces the growing perception that universities offer no value to students beyond credentialing. What is an in-person exam even testing? The skills that allow a student to do well on such an assessment task—memorizing, regurgitation, technologically unaided correct spelling and grammar, quickly drafting a passable response without feedback or revision—are completely irrelevant to the world we now live in. There are virtually no jobs, including being an academic, where these skills are important.

    Finally, we need to adapt to a world where people are increasing using AI. Yes, we must oppose the irresponsible offloading of important cognitive work to AI. But we must also recognize that there are constructive ways of using AI tools without abdicating intellectual responsibility. The coursework and assessments we set for our students should reflect this.

  5. The combination of students who are told that the only point of university education is to get a (better) job as quickly as possible and the widespread availability of AI – Oxford just made the best Chat GPT model available to all students free of charge, no doubt as a pre-cautionary measure – that makes this possible without much effort will destroy, if goes unchecked, not just universities but lots of human culture beyond.

    On a practical level, I still think there is no ‘solution’ to this situation other than humans realizing that this is not the way to progress. In the meantime, although for reasons of bias and practical challenges this no doubt has a bad pedigree in English-speaking education, I tend to combine written work with oral examination (I prefer to call it discussion). In-class exams, as was already pointed out in this thread, offer at best a partial and very restricted solution to the problem. No doubt bringing back oral discussion also doesn’t solve it. But it’s better than nothing.

    There can be other additions. Students can work on projects together, which then they have to present in class. Students can get weekly position papers, which then they have to defend in class. And so on. All these are of course massively time consuming, bias-inducing, and are pretty much unavailable in the context of large undergraduate education.

  6. When I began to work at Otago 37 years ago, the standard assessment regime for undergraduate courses was a mix of two take-home essays, in-class exercises (such as a non-mandatory presentation) and a final exam, with at least half the mark going on the final exam. This involved writing three-to-four essays in three hours. The exam questions were related to the essay questions and the topics covered in class, BUT the convention was that we would devise the exam questions with a slight twist, forcing students to think and thereby catching out the mere regurgitators. Typically, the last class of the term would involve a sermon on exam prep and exam technique, explaining to the students that we might be asking them questions with a twist (though not what the twists were going to be) and that they were going to be assessed (in part) on the relevance of their replies. I even had my little mantra: ‘The first rule for exam success is ‘Answer the question!’. The second is ‘Answer the question!’. And the third, which is really important, is ‘Answer the question!!!’
    I gradually moved away from this system in favour of a regime based on three take-home essays and a relatively small in-class component involving a (sometimes optional) student presentation. In effect, I was replacing an exam, which involved writing three essays in three hours, with a third take-home essay worth a much lower percentage of the final grade (the percentage mark on the first two take-home essays being expanded). The students would thus have the leisure to produce a much more considered piece of work and the process would be a lot less pressured. Moreover a take-home essay is ‘formative’ since you learn while writing, but an exam answer is merely ‘summative’ since you can’t write your answer unless you have already done the learning. I had both educational and self-interested reasons for implementing this change. The educational reason was that, in my opinion, the only advantage of exams over take-home essays is that exams are cheat-proof and take-home essays are not. Since cheating did not seem to be a major problem, the educational advantages of an extra essay trumped the cheat-proof advantage of a final exam. The self-interested reason was that I detested the chore of deciphering thirty or forty multi-page student scribbles at the end of every semester (sixty to eighty if I was teaching more than one course). It was much less disagreeable to work my way through thirty or forty well-formatted and neatly-typed essays.
    The ease of cheating with LLMs is going force me to revert to something like the former system.
    Fifty to sixty percent of the final grade will go on the exam, which will be a three-essays-in-three-hours affair. If low-cost undetectable cheating is rife then most of the assessment should go on the cheat-proof mechanism, however undesirable it might otherwise be. Students will still get to do two take-home essays, though the percentage of the final mark that goes on the take-home essays will be much reduced. However, students will have an incentive (motivating to the more intelligent) to write the take-home essays themselves, since if they don’t process the material through their own brains, they are likely to come a cropper when it comes to the exam. I can’t stop them cheating, but the large percentage of the final mark on the exam means that cheaters will be less likely to prosper. For me, the pedagogic costs of reverting to the former system will be low since Otago has a centrally administered system for running and invigilating exams. I won’t need to take time out from classroom teaching to put it into practice.

    But that still leaves the chore of deciphering student scribbles, made worse by the fact that present-day students are not in the habit of hand-writing. I think I have a solution to this problem though not one that is likely to become available to me during the last three years of my teaching career (so I shall just have to suck it up). It’s an idea that would have to be implemented by universities or divisions rather than departments and individual teachers, so if it is a good one, there would have to be a ‘march through the institutions’ to put it into effect. Universities could purchase basic (and hence cheap) ‘exam’ computers with no internet access and nothing more mounted on them than standard word-processing programs and other add-ons for non-philosophical subjects such as spreadsheets and statistical packages. The students type their responses into these computers under exam conditions. The responses are then either printed out or made available to the the marking staff to view online. The computers could then be blanked and made ready to the next round of examinees. How exactly to do this, I am not at all sure. Perhaps course coordinators could set exam questions in Word documents to which examinees would respond WITHIN those documents, the documents themselves to be mounted on flashdrives which would be collected by the invigilator at the end of the exam. More computer-literate readers can probably think of better ways of achieving the same result. However SOMETHING along these lines would be good for BOTH examiners, who would not have to pore through reams of barely decipherable scribble, AND examinees, most of whom are probably a lot better at typing than writing.

    But although the are things we can do to mitigate the problem, the LLM’s have dealt a severe blow to essay-based assessment and education, forcing us to revert to less desirable methods of teaching and assessment, some which involve a lot of extra work. Is there any possibility of SUING the manufacturers of LLMs for trashing the world’s education systems and making our lives more miserable?

  7. I don’t see a hard problem for education in AI.* It will force everyone to move to a system where individual professors look individual students in the eyes and talk about ideas face to face. Intellect and ingenuity cannot be faked or hidden looking another person in the eye.

    Unless we want to become the society from Star Trek that gives up our intellects to let the Master AI run everything (which we don’t want!) there is a deep incentive to continue the passage of knowledge and tools for its production from person to person. After all, in every organization there still needs to be people who know how to do the thing the organization does.

    Those who freeload off LLMs will plausibly end up without a white-collar job, because those jobs—the knowing how to do the basic thing, or at least how to moderate the AI that does the thing—will be what are left. There will be fewer white-collar jobs that just involve using tools without knowing what is really going on, because those will be automated.

Designed with WordPress