Leiter Reports: A Philosophy Blog

News and views about philosophy, the academic profession, academic freedom, intellectual culture, and other topics. The world’s most popular philosophy blog, since 2003.

  1. Brian W.'s avatar
  2. Steve Diamond's avatar
  3. Brian Leiter's avatar
  4. Phil Bold's avatar
  5. Gerald Postema's avatar
  6. current graduate student's avatar

    This is an interesting post. One important consideration that is not mentioned is that outsourcing RA-style work to LLMs, rather…

  7. Mark's avatar

A survey of academics about their AI use…

…from Oxford University Press.

Does this ring true to readers based on their experiences?

,

Leave a Reply

Your email address will not be published. Required fields are marked *

12 responses to “A survey of academics about their AI use…”

  1. The first thing to note about this study is that it took place in April, so five months ago. I went back and read some notes and blog posts of mine from five months ago, and they feel like dispatches from a completely different era (because they are).

    I’m grateful for this report, but readers should keep firmly in mind that it is badly outdated. To take just one example, I strongly suspect (based on my experience both with the tools and with academics) that fewer people are saying “I don’t use AI because its outputs simply are not useful.”

  2. Commenting to subscribe to this. Ring of Gyges thought experiment enthusiast, here.

  3. I was a respondent, and probably fell into OUP’s “persistent skeptic” category. I don’t know anyone who uses AI in research. I do know some who use it for administrative tasks, and I’m thinking of using it for similar things myself (e.g., converting a text-based bibliographic database into a spreadsheet format). I exclude from “AI” things that are merely clever and convenient, and have long been used (e.g., using Photoshop to assemble composite figures for publication).

  4. As Nate notes, the study predates Claude Fable 5, which to me is when consumer LLMs became useful for research.

    For anyone who still finds these tools useless, here’s a simple test: take one of your own drafts, give it to a current frontier model (Claude Fable 5.1 or GPT-6 Astra) with the reasoning effort set to maximum, and ask for a mock peer review. You might be surprised by the result.

    For me specifically, as a non-native speaker, having unlimited access to an instant proofreading service has also been a real boon.

    There may be valid ethical or privacy concerns about using these models, but I doubt lack of utility is still one of them.

    1. Fable *was* transformative (in a good way).

      Another trick — depending on your risk profile — which makes AI much more useful is to use Claude Cowork, or Claude Code from the terminal, and switch on “Auto” mode, so that it can read and write files. It requires trust, but it enables new workflow patterns which are not available with chatbots.

      For example, I hand-selected a list of 10-15 articles which featured computer simulations in formal epistemology and told Claude to (a) search the bibliographies of all of those articles, (b) for any article which it judged to be relevant for my project, add it to my bibliography database if it wasn’t already there, (c) track down a PDF of that article from the internet, (d) link that PDF to the database, then (e) continue with the bibliography search recursively until it felt it exhausted all the reasonable candidates from philosophy and philosophy-adjacent areas. It ended up adding over 300 files (most with PDFs) to the bibliography database. Once it did that, I then asked Claude to validate all the downloaded PDFs because some of them were misfiled / mislabelled — so it “viewed” the first few pages of each PDF and manually OCR’d them to ensure that the PDF matched the bibliography article it purported to be for. It then read and generated a detailed executive summary of each article (not based merely on the abstract) and wrote that into a LaTeX file, which compiled into a 200+ page PDF, with links so that I can read the summary and open the article with a click.

      Claude did that almost entirely autonomously, in the background, over a couple hours. Will it have made mistakes? Yes. Is what it did still genuinely useful and a huge time savings over me doing the same task by hand? Absolutely. The more logistical and operational matters I can offload to agentic AI, freeing up time to read and think, the better.

      1. In primarily theoretical philosophy and mathematics, the goal of most practitioners is perhaps primarily understanding. It is attractive to believe that someone who spends years finding such sources organically and gradually building up a grasp of the state of the field through intentional discovery of existing findings has a better foundation for gaining understanding through the course of her research than someone who delegates the entire arc of source discovery to an AI agent.

        1. That’s a fair worry, but I think it describes a more wholesale delegation than what I did, and I also think that it proves too much: that way of thinking also rules out any use of research assistants. It also appeals to a certain ascetic, and aesthetic, ideal of the individual researcher of which I’m a little suspicious.

          It sounds a bit like an updated-for-AI version of the argument, which people made in the early days of Google, that students should be discouraged from using it because it robbed them of the serendipity of browsing stacks of books in the library. I don’t think that argument has fared particularly well.

          It’s worth focusing on what that agentic workflow did: it followed bibliographies. That’s exactly what most people do when they build up a grasp of a literature “organically” — you start from papers you trust and chase the references. The difference is that Claude did it exhaustively in a few hours rather than partially over weeks or months.

          You are right that much of the intellectual value of research comes from “the intentional discovery of existing findings” and the understanding that brings. But the information landscape has changed enormously since the days of Poincaré and Hilbert, arguably the last mathematicians who grasped most of the frontier of their field. The scientific literature has grown exponentially — the large majority of all scientific papers ever published have appeared in the last fifty years — and that creates a new problem for academics.

          Understanding is achieved through reading and thinking, writing and reflecting, discussing, arguing, and trying to apply the ideas we have — but the ideas we encounter are enormously path-dependent. One of the greatest issues with the contemporary information landscape is the problem of unknown unknowns. I can’t learn from a paper I don’t know exists. And, given the sheer number of papers, how can I separate the wheat from the chaff and figure out what to dedicate my limited time to? Yes, we had a number of standard methods in the past: rely on the recommendations of friends, conferences, author prestige, journal prestige, accidental serendipitous discovery through whatever means. Those were reasonable solutions, at the time, for boundedly rational agents like ourselves because those solutions were pretty much the only solutions available. That doesn’t mean they were the best solutions.

          Maybe you were responding to my remark that I told Claude to continue “until it felt it exhausted all the reasonable candidates”? What I meant, there, was this: a lot of papers in philosophy cite other philosophy papers, and then make connections to further-afield interdisciplinary literature. Some of those further-afield links will be interesting and relevant but, a lot of the time, they are cited because the author needs to show familiarity with the field. A full recursive bibliography search would quickly risk pulling in all of academia. If one of the neglected sources turns out to be important, I’ll see that source when I read the paper and — if I find it interesting — I’ll be sure to follow up.

          All I was saying, in my previous post, is that agentic AI provides a set of tools for cutting out the dogsbody work of finding what to read and getting hold of it. There’s no intellectual benefit from the amount of time it takes me to type up a BibTeX entry in a database, download the PDF, or look up the ISBN to fill out an interlibrary loan request, and so on.

      2. Please correct me if I’m simply in error or being grumpy & unfair, but I find this a bit hard to understand, really.

        By your own lights, Claude is likely to have made mistakes. (Actually, you say “it will have,” suggesting you know it has, not simply that you suspect it might have.) So, you shouldn’t defer to Claude’s summaries in any serious way without double checking their accuracy (and, presumably, coverage), unless you have very good reason to know exactly what it is likely to have gotten wrong where, which would seem to presuppose knowledge of the literature you’re attempting to gain familiarity with. So, assuming you shouldn’t defer to Claude’s summaries in any serious way without double checking, and you want to defer to Claude’s summaries — why else have the little guy make ’em? — you oughtta double check. It seems unlikely to me that double checking ~300 papers and comparing them to Claude’s summaries will be much of a time saver, if any, over just reading the darn things (or, more realistically, a small but usefully representative subset thereof).

        There’s also the separate issue, which “Santa Monica” raises, about whether deferring to and working with Claude summaries (even if it were permissible/professionally responsible) is unlikely to provide one with the kind and depth of understanding serious philosophical work with a literature requires.

        So, I’m not seeing a good/sufficient reason to follow suit/adopt your practices. It seems like you probably think there is one, or at least that there’s good enough chance there’s one for someone like me to take the possibility seriously. Could you help me see things your way?

        1. Ah! I meant “very good reason to think you know” in the second paragraph, alongside “likely” in the third paragraph.

        2. Thanks for the clarification!

          I wasn’t saying that I *defer* to the summaries, in the sense of taking them as accurate representations of the papers, but rather treating them as guides for what to read. As I mentioned in my reply to Santa Monica, the exponential growth of publications creates a real problem of how to decide what to read, and how to find what to read. The AI summaries provide a new way of doing that, given that any person is only going to read a subset of the 300 papers found in the search. The point is, whatever subset is selected, I then read in full — just like before AI existed.

          Claude makes mistakes, but everyone makes mistakes. Your best friend who recommends a paper will, on occasion, make mistakes. The question is whether the signal-to-noise ratio is sufficiently high that the summarisation work from agentic AI provides a better starting point than going in cold yourself. This especially matters given the problem of unknown unknowns.

          How do you decide whether to read a paper? You’ll look at the title and then the abstract. But realise that both the title and the abstract are written by a party (the author) who has a vested interest in making the paper sound as interesting as possible in order to make you want to read it. There’s no way to double-check an abstract without reading the paper, but we still rely on abstracts! If an AI agent reads through it and suggests that the abstract over-sells the paper, that’s something important to keep in mind.

  5. I’m a mathematician, not a philosopher, but in case people are interested: I use AI a lot for information retrieval. I find deep research to be invaluable. I do *not* trust the AI’s summary of the research; I use it as a superpowered version of search and actually read the links. I would really like to have a version of this that could be hooked up to academic databases. I’m sure such things exist but I’m not at a university, so the best I can do is tell the agent to limit itself to peer-reviewed and academic sources. On a smaller scale, I also find chatting with Claude to be a great way to look up bits of code syntax – e.g., how do I format this table in LaTeX this way, how do I get matplotlib to look like this. Google search used to be great for that, but the proliferation of spam sites out there, and the death of Stack Overflow, makes it easier to just ask Claude.

    I use AI a lot for review. I’ll have it check my code to see if there are ways to improve runtime or find bugs. I’ll ask it to review papers. I find its feedback genuinely valuable, especially for code – it’s cracked bugs that I was stuck on and suggested significant optimizations that I hadn’t thought of. Nothing superhuman, but still quite useful.

    I *rarely* ask AI to actually *write* code, because most of the code I write is experimental code. It’s not just about correctness, I need to make sure that *I* understand it, and I find it easier to write it myself than to review AI-generated code thoroughly enough to achieve that understanding. I’ll ask it to write code in cases where that doesn’t really matter – e.g., bits of HTML formatting for my website – or where it’s trivial.

    An exception to the above: I have experimented with AI-based optimization loops. Non-convex optimization is very difficult and there have been reports that AI is really good at it. For this particular problem, it also didn’t really matter *how* the parameters were generated, it only mattered that they *existed*, so I was okay with not fully understanding the process. However, AI failed to improve on my merely human baseline in this particular case.

    I *never* use AI to *write* papers, emails, comments, or other communications to humans. I’ll ask it for feedback, and it often gives good feedback, but I consider it a point of honor that all words should be my own.

  6. If it’s useful to hear a perspective from an independent publisher who has worked at the major commercial firm as well as a major university press, it makes very little sense to talk about AI in this holistic way. Far too much depends on the specific disciplines and kinds of papers that are being written. A medical clinical trial paper, a review of recent diabetes papers, a jurisprudence paper in a peer-reviewed journal, a jurisprudence paper in a law review, and an ecologically-inflected take on Shakespeare’s The Tempest are all going to raise very different AI concerns and be open to varied AI opportunities.

    If a paper is based on a lot of empirical data that has to be analyzed statistically in different ways, then AI can be very useful.

    If you do need to provide a thorough review of recent literature, then it can also be very useful, but vigilance is particularly important here. I can recommend Magnus as a good tool for references, incidentally.

    At this time, the policies of journals are generally hostile to substantive forms of AI writing, but more basic editorial assistance functions (like formatting, spelling) are becoming acceptable as long as they are disclosed.

    If most of the paper is prose that is interpretive, rather than the presentation of an experiment or heavily “data-driven,” then the whiff of non-human composition is going to be the kiss of death.

    That could change eventually — Luciano Floridi just had an entire book written by AI as an experiment in “distance writing.” If that becomes the norm, then civilizational collapse is not far behind. Stanislaw Lem, no Luddite, believed this back in 1981 when he wrote his “Golem XIV,” to my mind one of the most important discussions of AI, putting much of the stuff written in the past 4 yrs in the shade.

    The AI policies of every journal/publisher need to be closely studied in order to determine what is not acceptable.

    As many fellow readers will know, there is disturbing hypertrophy afoot in the area of journal retractions. Thousands of papers are being retracted every year because of what is called “research integrity.” Every publisher is desperate to avoid the increasingly public black eye that comes retractions, especially in the form of falsified or fraudulent research.

    This sad truth is that much of this takes place before the research ever reaches the publisher. Peer review can be handled badly, just like any human process. But the bad data comes from the author(s). And not every retraction is, of course, an ethical failure. But that is a relatively rare nuance in a field that is increasingly hostile to the finer distictions.

Designed with WordPress