Leiter Reports: A Philosophy Blog

News and views about philosophy, the academic profession, academic freedom, intellectual culture, and other topics. The world’s most popular philosophy blog, since 2003.

  1. Brian W.'s avatar
  2. Steve Diamond's avatar
  3. Brian Leiter's avatar
  4. Phil Bold's avatar
  5. Gerald Postema's avatar
  6. current graduate student's avatar

    This is an interesting post. One important consideration that is not mentioned is that outsourcing RA-style work to LLMs, rather…

  7. Mark's avatar

More on AI doing history of philosophy

I wrote in September about a high-level LLM’s critique of a draft paper of mine (“Nietzsche on Creating Values”) on which I had also gotten feedback from serious, philosophically-minded Nietzsche scholars. Subsequently, philosopher Jason Alexander (LSE) offered to run the paper through another high-level LLM (to which he subscribes), but first he uploaded the entire Nietzsche corpus to the LLM and then asked the LLM to prepare a referee report. The referee report was about 50% longer than the paper. It took a lot of time to get through it, and while in some ways it was impressive, it was not a good referee report (as the length might suggest). The LLM was excellent at finding every reference to a key word in the corpus, but it was much less good at figuring out which ones were really relevant to the argument or the concepts in play. It did identify two passages about which I will add some comments in a footnote, so that was useful. The LLM referee report criticized me for not mentioning several pieces of secondary literature, although in each case I didn’t mention it for a reason: I’m not writing a literature review, and this literature is either not really relevant or just not very good, so one should pass over it in silence.

In both cases, the key problem is the LLM has no “judgment” (as I said to Professor Alexander). It performs RA-style tasks comprehensively, but then one has to sort through the results. It is quite good at fixing citations or catching misquotations–again, something you might ask an RA to do. I will acknowledge both the LLMs in the final version of the paper.

On the lack of judgment point, Professor Alexander had some interesting thoughts, which he kindly gave permission to share:

I’ve spoken to a couple of people in my department about how to make sense of the current (mixed) state of AI to do advanced philosophy, given its remarkable ability in solving programming and mathematical problems.  I think there are several factors at work, here. First, it’s clearly the case that all the AI companies are doing heavy reinforcement learning to maximise performance and accuracy for programming, mathematical and other technical subjects, because that is their business model. I find it interesting that LLMs still under-perform in areas like law and accounting, where the rules-based nature of the field would, at first impression, seem to make it more amenable to LLMs.  But, if this is the case, I find it remarkable that they can do any philosophy *at all* since it’s not an area of optimisation.

Second, I suspect that one of the problems LLMs face with doing advanced philosophy well is that much of the “knowledge” required to do so is either tacit knowledge, or “know how”, or else socially distributed knowledge that *isn’t written down* and so LLMs can’t be trained on it.  With a few exceptions (e.g., Kerry McKenzie’s magnificent take-down of Colin McGinn’s book), a philosophy journal won’t publish a paper which says “so-and-so wrote a really bad article that is utterly incompetent and moronic in its analysis of X”. Instead, that assessment will be communicated orally in seminars and will be reflected in the field by articles vanishing from the conversation. An LLM won’t be able to use that information to determine if an article vanished because it was bad or it vanished because it was overlooked.

Maybe the best way to get advice from an LLM would be to prepare a background brief B saying what fundamental assumptions should be unquestioned, what theoretical angle should be assumed when working, what primary texts should be relied on, and what other texts / conversations should be disregarded. I bet an LLM would do a pretty good job, given such a B. The problem with an LLM is that, in the absence of such a brief, it is effectively coming up with the “best possible answer” for all possible Bs.

That’s probably right, but raises the question whether it is worth the time to do that, or whether it’s better to use the LLM for the RA-style work, and simply get feedback from experts in the field, as we have long done. (Professor Alexander’s observations do underline the importance of critical book reviews, and I am certainly trying to do my part in my fields!)

, , ,

Leave a Reply

Your email address will not be published. Required fields are marked *

7 responses to “More on AI doing history of philosophy”

  1. I just wanted to post a quick follow-up regarding the methodology used for the experiment Brian described in the original post.

    This was done using Claude Code, Fable 5.1 on the Ultra plan. I downloaded public domain versions of all Nietzsche’s works from nietzschesource.org and saved the content in plain text files on my laptop. Between the Werke, Nachlass and Briefe, that amounted to 2,669,057 words, or somewhere between 5 to 7 million tokens.

    Using Claude Code was essential, here, because it would not have been possible to have the AI agent access all the content, since Anthropic’s publicly available agents have a maximum context window of 1 million tokens. However, since the files were saved on my laptop, the agent could use standard command-line text-searching tools on my laptop (sed, grep, etc.) to do localized searches, only reading the relevant portions into its context window, as needed.

    I also broken the task down into multiple agents running in parallel, each tasked with searching part of the Nietzsche corpus, with each agent writing its findings into a summative brief, with all the summative briefs going to a single Fable agent, who put all the pieces together.

  2. “…the key problem is the LLM has no “judgment”…

    I am calling that a blessed feature.

    Judgment, if Kant is to be believed, is a distinct mode of cognition. AI can simulate mereological judgments but does not perform them.

    If we are judging the work of AI strictly in terms of outputs, simulation of mereological judgment should be sufficient, but the future of academic philosophy lies in clarifying this simulation is not remotely sufficient or even intelligible. We cannot save the enumerative forms of knowing but judgment is not an enumerative act.

  3. > I will acknowledge both the LLMs in the final version of the paper.

    This is something I’m struggling with – whether and how to acknowledge LLM use, since norms on this aren’t really settled. (Just for clarity, I *never* let LLM *write* for me – all words are always my own.)

    > But, if this is the case, I find it remarkable that they can do any philosophy *at all* since it’s not an area of optimisation.

    This gets at a really fascinating question, and one with critical implications for whether “superintelligence” is even possible: how do LLMs generalize from problems seen in training to novel problems? It’s clear that their ability to generalize isn’t unbounded, as discussed above. At the same time it’s clearly not zero, or else they wouldn’t be able to solve novel problems at all. I don’t think we even really have the right way of formulating the question yet.

    As far as LLMs doing history of philosophy: my uses are much less advanced than yours, since I’m not a trained philosopher, but I do want to throw them in the pot for consideration. I frequently use LLMs to generate reading lists on philosophy. For example, I’ve been doing a lot of reading on analytic epistemology of understanding and wanted to broaden my reading to include anything analogous on the continental side, and I had Gemini generate a reading list. Since I’m not at a university, there isn’t really a human philosopher I can ask these questions. I don’t have the knowledge base to really evaluate it, but I’ve been pretty happy with what it comes up with as a starting point – certainly better than just Google searches or reddit.

  4. current graduate student

    This is an interesting post. One important consideration that is not mentioned is that outsourcing RA-style work to LLMs, rather than having an actual RA or graduate student advisee do it, deprives that RA/grad student of the opportunity to learn a literature under the guidance of an experienced advisor. These sorts of practices constitute one crucial way that the “know how” of a field/literature is learned in the first place. This point mirrors other professions where AI’s first casualties are entry-level positions, a situation which will eventually create a dearth of experienced workers.

    1. To be clear, I will continue to use human RAs for most of the RA-style tasks, since the LLM, as I wrote, gives too much information without much discernment. How much value this kind of work is for graduate students is another matter (apart from monetary value, which is important) is less clear.

  5. If the goal of these companies is to produce “artificial general intelligence” or “artificial superintelligence,” then they’d eventually have to solve the problem of creating a model with good judgment. But if they set their sights on that goal, I’m not sure our discipline will survive, much as the focus on mathematics has been detrimental (by many lights) to that discipline.

    Much of the social knowledge, etc., has been, can be, or will be written down. So I don’t see that as being an a priori obstacle to producing models with “good judgment.”

    I’d like to be less pessimistic. But whenever I hear things along the lines of “Surely AIs won’t be able to do X, Y, or Z!” the word “surely” rings hollow. (Of course, for claims like “Surely they won’t have souls, consciousness, agency, or whatever,” there is, at the moment, no consensus on whether we could tell if they “really” had these things, especially as they become more advanced.) Happy to be convinced otherwise!

    1. As for the claim, “Much of the social knowledge, etc., has been, can be, or will be written down.” – I am reminded of the essay, “I, Pencil” by Leonard Read. That essay points out how no single human has complete knowledge of how to make a pencil. I suspect that the knowledge has not even been written down, at least not in its detailed entirety. For that matter, there are illiterate humans with generations of knowledge that cannot be written down because they can’t write.

Designed with WordPress