Leiter Reports: A Philosophy Blog

News and views about philosophy, the academic profession, academic freedom, intellectual culture, and other topics. The world’s most popular philosophy blog, since 2003.

  1. Prospective Applicant w/ Top 15 Philosophy B.A.'s avatar

    If the GRE is required (and I am skeptical that it should be), I think it’s important to consider 1)…

  2. Public R1's avatar
  3. JR's avatar

    …and the GRE doesn’t differentiate at the top as clearly with the new-ish scoring. So it may not be as…

  4. JR's avatar

    The GRE & the SAT are nowhere near as g-loaded as they once were. Neither do they compare to WAIS…

  5. Mike Otsuka's avatar
  6. Russ Winner's avatar
  7. Mike Otsuka's avatar

More on AI doing history of philosophy

I wrote in September about a high-level LLM’s critique of a draft paper of mine (“Nietzsche on Creating Values”) on which I had also gotten feedback from serious, philosophically-minded Nietzsche scholars. Subsequently, philosopher Jason Alexander (LSE) offered to run the paper through another high-level LLM (to which he subscribes), but first he uploaded the entire Nietzsche corpus to the LLM and then asked the LLM to prepare a referee report. The referee report was about 50% longer than the paper. It took a lot of time to get through it, and while in some ways it was impressive, it was not a good referee report (as the length might suggest). The LLM was excellent at finding every reference to a key word in the corpus, but it was much less good at figuring out which ones were really relevant to the argument or the concepts in play. It did identify two passages about which I will add some comments in a footnote, so that was useful. The LLM referee report criticized me for not mentioning several pieces of secondary literature, although in each case I didn’t mention it for a reason: I’m not writing a literature review, and this literature is either not really relevant or just not very good, so one should pass over it in silence.

In both cases, the key problem is the LLM has no “judgment” (as I said to Professor Alexander). It performs RA-style tasks comprehensively, but then one has to sort through the results. It is quite good at fixing citations or catching misquotations–again, something you might ask an RA to do. I will acknowledge both the LLMs in the final version of the paper.

On the lack of judgment point, Professor Alexander had some interesting thoughts, which he kindly gave permission to share:

I’ve spoken to a couple of people in my department about how to make sense of the current (mixed) state of AI to do advanced philosophy, given its remarkable ability in solving programming and mathematical problems.  I think there are several factors at work, here. First, it’s clearly the case that all the AI companies are doing heavy reinforcement learning to maximise performance and accuracy for programming, mathematical and other technical subjects, because that is their business model. I find it interesting that LLMs still under-perform in areas like law and accounting, where the rules-based nature of the field would, at first impression, seem to make it more amenable to LLMs.  But, if this is the case, I find it remarkable that they can do any philosophy *at all* since it’s not an area of optimisation.

Second, I suspect that one of the problems LLMs face with doing advanced philosophy well is that much of the “knowledge” required to do so is either tacit knowledge, or “know how”, or else socially distributed knowledge that *isn’t written down* and so LLMs can’t be trained on it.  With a few exceptions (e.g., Kerry McKenzie’s magnificent take-down of Colin McGinn’s book), a philosophy journal won’t publish a paper which says “so-and-so wrote a really bad article that is utterly incompetent and moronic in its analysis of X”. Instead, that assessment will be communicated orally in seminars and will be reflected in the field by articles vanishing from the conversation. An LLM won’t be able to use that information to determine if an article vanished because it was bad or it vanished because it was overlooked.

Maybe the best way to get advice from an LLM would be to prepare a background brief B saying what fundamental assumptions should be unquestioned, what theoretical angle should be assumed when working, what primary texts should be relied on, and what other texts / conversations should be disregarded. I bet an LLM would do a pretty good job, given such a B. The problem with an LLM is that, in the absence of such a brief, it is effectively coming up with the “best possible answer” for all possible Bs.

That’s probably right, but raises the question whether it is worth the time to do that, or whether it’s better to use the LLM for the RA-style work, and simply get feedback from experts in the field, as we have long done. (Professor Alexander’s observations do underline the importance of critical book reviews, and I am certainly trying to do my part in my fields!)

, , ,

Leave a Reply

Your email address will not be published. Required fields are marked *

Designed with WordPress