News and views about philosophy, the academic profession, academic freedom, intellectual culture, and other topics. The world’s most popular philosophy blog, since 2003.
If you were the head recruiter for a professional basketball team, would you require height and weight measurements for prospective…
I wonder if people are familiar with Fred Schauer’s essay on Kelsen and Fuller. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3607099
If knowledge is sociologically conditioned, then what, exactly, happens to the truth-status of the sociology of knowledge itself? This is…
To a large degree, Kuhn’s Structure is an attempt to deliver on the descriptive task of epistemology as Reichenbach conceived…
I concur to a certain extent. I also scored very low on the writing section. Back then, GRE was required…
Thanks, I’ll check it out!
I’m a graduate student in philosophy at a well-known terminal MA program who is applying to PhDs this fall and…
First, the worry was that students were using it to write their papers; now it turns out instructors are using it to evaluate them! What do readers think? Have any of you used AI this way?
I retired before this monster made its appearance in academia.
Sorry for the curt response to this phenomenon, but any professor who uses AI to assess student papers should be fired.
Is that too harsh? Strictly illegal? Both? Neither?
For most teachers of philosophy, a big part of our job is convincing our students that the texts and topics we're asking them to read and write about matter. For most student writing, the only real audience is the professor (or TA/grader). If we show students that we don't care enough about their writing to take the time to read it, how can we expect them to care about what we're asking them to write?
The only good thing about this example is the role it can serve in a reductio of the all-too-common suggestion that AI is a tool that we should be teaching students how to use (a suggestion made regardless of the level or goals of the course). If it corrupts the grading process (as I believe it does), then it certainly corrupts the goals of most valuable assignments (especially in philosophy and courses that teach you how to think).
If this actually becomes a thing among pedagogues (and just taking into account human nature per se, it seems highly probable), then it seems only fitting to dust off the hoary quote: Quis custodiet ipsos custodes?
One can already envision a matryoshka-like recursion or 'infinite loop' of evaluators using AI to evaluate their subalterns every descending step of the way and vice versa in reciprocal ascending order.
Mutatis mutandis for students and teachers, etc, and so, with its own 'phenomenology of addiction'/'addictive phenomenology' to boot: everything must be filtered through the AI linguistic machine (the worm at the core of the apple that takes over the brain), becoming both (anti)-heuristically and epistemically obligatory, but—here's the kicker—with an acute ludic buzz, a la playing the slot machines.
In other words, a dynamic, on the whole, conspiring to feed the AI linguistic/imagistic/ sonic, etc machine oodles and oodles of human externally expressed thought (and the indivisible psychological residue contained therein, like a partial phonograph), so that in seeking the perfect likeness of man, we may inadvertently come to perfect the wolf within along the way.
It's tempting, because I see _so_ much AI work. It seems only fitting to respond in kind, the ethics of it all be damned. But I strongly suspect it would (1) take more work on my part, and (2) leave me feeling even more alienated than I already do.
And that's the sticking point, really. When 50% or more of the work I mark is clearly AI-generated, it just makes me feel like I've wasted all my time. Not just the time spent reading and marking, though that's significant, but the entire semester's worth of building connection with the students. If that's how they're going to engage with _me_–i.e. by not engaging, and instead transparently lying… Well. I guess I feel used as a mere means to someone's ends!
You can see why the thought of just outsourcing it to the machine is attractive. But it's also just not going to make me feel any better. I can't control what they do, but I can at least still uphold the standards I apply to myself. I may as well not treat _myself_ as a mere means!
Do I use AI to grade assignments? No. Am I worried about students using AI to do their assignments? Yes. But I'm not worried about them using AI to kick-start the process, though in the class I'm teaching this semester–legal research–we have prohibited even that. One reason for the prohibition is that in order better to understand AI one needs to have a sense of the kinds of material on which the model is trained. Last week we presented a guest speaker in our class, Pablo Arredondo, who was a leader in the development of the first legal research service driven by GenAI, CoCounsel, which was recently acquired by Thomson Reuters for a goodly sum. He is well aware of the risks, and he described how CoCounsel is designed to mitigate them. But he is particularly concerned about the wrong uses to which he expects it will be put, in particular as a substitute for writing that requires creative expression. Compare, for example, drafting a simple declaration versus preparing a Motion for Summary Judgment. He worries that law students will forgo opportunities to draft complex documents and by doing so they will lose an important lawyerly skill. Still, AI is already showing promise in practices that aren't as resource rich as, say, Big Law, such as Access to Justice services. These services don't have large staffs or massive revenue sources. They can turn AI on tedious, time-consuming work, such as document review, leaving human intelligence to focus on developing litigation strategy, crafting arguments, and meeting with clients.
I have no idea what kinds of assignments ("essays," says the article) one gets in strategic communications, but I would imagine that they resemble complex legal writing more than document review. That would argue against even "a first pass" by AI, rather than a graduate student. But it depends on the model and the degree to which it has been tailored to the particular task, i.e., not just straight out-of-the-box GPT4.
Maybe we should *try* grading assignments with AI, before we start all the doom & gloom. Let's start with some threshold questions:
1. Does it work in terms of determining which papers are better than others?
2. Can it generate useful feedback?
For the sake of argument, assume the answers to those questions are "meh, it's kinda rough, but Chat GPT 5 will probably fix that."
Then we can move on to some other questions like:
3. If it can grade better than I do–and generate better comments for students than I'd provide–what's the point of me grading? Say I could take that grading time and go volunteer in a soup kitchen. Say I could find other ways to invest it in my students.
Like this is sort of how I think through the decision tree–which isn't to say I have answers, just that this is the structure of my thinking. I really bristle at "fire them!", without any sort of empirical investigation into the usefulness and maturity of the technology.
Should we fire people who use voice-to-text? Should we fire people who use other novel technologies? What's so flipping sacred about *grading* of all things, that makes it untouchable vis-a-vis technology?
Respecfully submitting a different view. 🙂
I think these questions (from Fritz Allhoff) are very much the ones that will need to be decided over time. Here are just a few observations of my own.
First, many scientific journals and their publishers view the use of AI as a form of plagiarism or another kind of breach of publication ethics. These are breaches of professional norms that have led to termination in the past — the very recent past, in particular. So what would be required for this to become acceptable in the future is some kind of redefinition of the use of AI that makes it akin to keyword searching, or using a calculator in mathematics. I'm not saying that these are close analogies, just that this is the kind of normalization that would have to take place.
Secondly, it's a real question in my mind whether it makes any sense to think of AI as it is now being in a position to be a "better grader" in a subject like Philosophy, where the exams are not multiple choice questions, typically. At least, as I understand it, grades are based primarily on essays with arguments that require a great deal of contextual understanding. This seems to be exactly the kind of thing that AI is NOT good at, as numerous articles have shown over the past year. Whenever it attempts discursive prose, it becomes highly repetitive and reliant on cliche. I think the recent Hegel example that Brian shared is an example of that, though it does have it's impressive aspects.
I know my comments are about AI as it is now, and not as it may become. I just find it curious that there is much reliance on the faith that it will eventually become fully sentient. That relies on a metaphysics that one does not associate with the godless libertarians of Silicon Valley, but rather with the major religions of "the Book." The AI adepts and acolytes all say "It's not there yet, but one day It will come." There are many committed atheists or agnostics in the AI space who view something like the Singularity as a plausible Messiah. Blessed are those who have not seen, and yet believe.
Finally, in order to use AI as it is now at all well, you need to invest some time and effort in understanding how to ask it the right kinds of questions or "prompts." That is time that could perhaps more profitably be spent in direct teaching. It is one more example of how these alleged "tools" become our masters and force us to conform to their requirements rather than vice versa. That might not be enough for an Apocalypse, but it does feel like an abnegation of a part of our humanity in a way that a tool like the wheelbarrow doesn't.
"What's so flipping sacred about *grading* of all things, that makes it untouchable vis-a-vis technology?"
Nothing sacred at all, but there's technology and then there's technology. Easy example: the internet, which for a brief, innocent moment c.1990 we believed would help us "reach out and touch" each other in ways that relieved us of our addiction to postcards. AI is similarly un-hammerlike, a tool that solves uncertain problems, but that produces externalities, problems that demand new tools, and so on.
What is the problem solved by AI that we have traditionally resorted to grading to solve? At this early stage, I'd say it's little more than the burden of grading. AI can generate a vague semblance of grading, enough to convince or nudge us to delegate or abandon our responsibility to it. Good for lazy us.
While I am not a philosopher, I am in a cognate discipline. If >50% of the work you see. is AI-generated, then the glaringly obvious issue is that you need to change your assignment structure.
I have philosopher colleagues who have moved to oral exams. This is obviously not feasibly in all courses (or in any courses at many institutions of higher ed), but the core insight remains: faculty need to adapt.
(And if changing how you assess students is too much effort in course redesign… well, there are plenty of far more accomplished-given-career-stage candidates waiting to take your position.)
JT:
This is an institutional-level problem, not a problem unique to me. When I started here, I was told to expect failure rates up to 75%. (Yes, that's nuts; I mention it simply to make it clear that the problems here predated ChatGPT, and are exacerbated by a variety of factors, including the fact that everyone here is paid per-course and courses get cancelled when enrollment drops below 27, which doesn't give instructors much room to manoeuvre or crack down.)
But you're free to believe that I'm lazy deadwood who doesn't deserve his (non-TT) job, if you like.
I'm with Fritz Allhof above. I don't think there's any principled reason why AI couldn't play a useful role in providing feedback on essays.
In some sense we have this already, with AI providing spelling and grammar feedback directly within most word-processing software, and many faculty using software that provides students a plagiarism report after they turn in papers, which serves as a sort of feedback regarding whether they did enough to paraphrase ideas that they had attempted to put in their own words. This feedback isn't perfect, but that doesn't stop it from being useful.
We're on the brink now of having AI that can provide feedback that is much sensitive to the content and organization of papers. My sense is that how well it currently does varies a lot from topic to topic, but I think that, at least on some topics, it already does well enough that it could be reasonable for teachers to use, and that on many other topics, including most topics that philosophy undergraduates write about, coming advancements will make it similarly useful, probably within the next decade.
It's worth emphasizing that the claim here isn't (yet) that AI gives *better* feedback than an expert human who spends a lot more time, just that AI feedback can be useful. It may help to bear in mind that real human experts will spend only a finite amount of time on each student paper, and out of practical necessity will refrain from offering some helpful suggestions they could have made. If AI allows professors to provide better quality feedback than *that*, then that's an improvement. And the nay-sayers here who want, for example, to fire professors for even touching AI have fallen far short of showing that AI can't be useful in this way.
It's also worth emphasizing that hardly anyone is (yet) claiming that AI, on its own, can currently provide better feedback than professors, on their own, do. Instead the most reasonable claim in this ballpark is that some combination of professor+AI efforts can typically provide better feedback regarding some paper assignments than professors typically do currently, at no greater cost to the professor. I don't think any of the naysayers here have come anywhere close to showing that that claim is false.
AI spelling and grammar feedback is old AI. It isn't "generative," which is what all the hoopla these days is about. Old AI was for quite some time mere science fiction, and the approach that has won the day based on neural networks was largely dismissed at its inception. Spelling and grammar comply with rules; original texts do more than comply with rules.
Of course the claim is that AI gives better feedback! It's better, at the very least, because it takes less time, but not necessarily because it reliably catches where human experts, whose very texts inform the model, fell short. But I agree that the "first pass" approach can be beneficial, and that it might reveal issues to which even experts are blind. Hence, I don't believe that quality varies from topic to topic. I believe it varies from task to task. AI excels at tedious, iterative tasks involving huge amounts of text. It does not excel at sonnets.
Experience teaches me that every coming advancement, especially those that are hyped, assures unpredictable retreats. AI right now is egregiously expensive and environmentally destructive. It's fun, neat, cool, in demand, Samuel Johnson's dog dancing on hind legs. I await the moment when AI generates effective letters of reference for job applicants.
With regard to Prof Rowan's claim, here's the (inevitable) sonnet written by ChatGPT about why generative AI can write sonnets as well or better than a human (the self-deprecating final lines, notwithstanding):
In realms where lines of code weave verses fine,
Artificial minds, with circuits vast and keen,
Craft sonnets that in eloquence outshine,
With patterns structured, flawless and serene.
Unfettered by the frailties of mortal woe,
They dance upon the canvas of the mind,
In worlds of logic, where thoughts freely flow,
Their symphonies of words are well-defined.
No shadow dims their intellect's bright glow,
No bias taints the beauty they create,
Each word precisely placed in perfect row,
No human error mars their polished state.
Yet though they soar where human minds aspire,
Their art lacks soul, the spark of human fire.
That's fun, for sure. But is it excellent? I'm curious, too, about the prompt(s) that produced the poem.
I appreciate the book-end of "fine" and "fire," the recurring "n" sounds, and the ongoing "o" rhymes. But then we have "dance upon the canvas," "symphonies of words," "polished state," all clunky doggerel.
Science reports on an effort to establish guidelines across the sciences for appropriate usage of AI in research: https://www.science.org/content/article/should-researchers-use-ai-write-papers-group-aims-community-driven-standards
Thank you, Robert, for the honorific, but I am not a professor, though I teach a class. Anyway, I've been a Dean longer than any of your campus leaders.
—–
KEYWORDS:
Primary Blog
Leave a Reply