Leiter Reports: A Philosophy Blog

News and views about philosophy, the academic profession, academic freedom, intellectual culture, and other topics. The world’s most popular philosophy blog, since 2003.

  1. Justin Fisher's avatar

    To be worth using, a detector needs not only (A) not get very many false positives, but also (B) get…

  2. Mark's avatar

    Everything you say is true, but what is the alternative? I don’t think people are advocating a return to in-class…

  3. Deirdre Anne's avatar
  4. Keith Douglas's avatar

    Cyber security professional here -reliably determining when a computational artifact (file, etc.) was created is *hard*. This is sorta why…

  5. sahpa's avatar

    Agreed with the other commentator. It is extremely unlikely that Pangram’s success is due to its cheating by reading metadata.

  6. Deirdre Anne's avatar
  7. Mark's avatar

AI cheating and pangram redux

Brian Leiter

Philosopher Stefan Sciaraffa at McMaster in Canada writes:

I read the Unherd piece…I align with its general  spirit. However there is one key claim that I’m not so sure about. I’ve been using Pangram. It has a vanishingly small false positive rate. Folks at  the business school at your university have verified this. I’ve run 100s of 2022 and earlier essays from my students without a single hit, corroborating the claims regarding virtually zero false positives.  This detector does show ai usage in my student’s papers for this year and last.  So I’m not sure about the linked article’s claim that the false positive rate is too high for all AI detectors. One possibility is that Pangram can read the date of the paper’s authorship. Hence, no false positives prior for 2022 and earlier essays. I can’t read Pangram’s code. So I can’t rule that out.  

Pangram came up on an earlier thread. Curious whether any readers know whether Pangram’s code allows it to detect the date of a piece of writing?

,

Leave a Reply to Justin Fisher Cancel reply

Your email address will not be published. Required fields are marked *

5 responses to “AI cheating and pangram redux”

  1. Mark

    There’s a simple way to test. Open a pre-2022 essay and copy-and-paste it into a new file.

    Reply
  2. Deirdre Anne

    I see this question as a bit naïve. There is metadata on every document created by a modern word processor and every pdf (unless it’s expressly stripped). This makes it trivial to determine the date of creation of any such document. It is possible to spoof this data, so if Sciaraffa’s concern that Pangram is reading this to game its own analysis were true, it would be likewise trivial to trick the system.

    An interesting test case might be to identify both pre-2022 documents and post-2002 documents, some of the latter specifically known to be written by GenAI, and then strip the metadata and upload them to Pangram.

    Reply
    1. Keith Douglas

      Cyber security professional here -reliably determining when a computational artifact (file, etc.) was created is *hard*. This is sorta why when your computer clock battery dies websites (which use TLS- the “s” in HTTPS) start going nuts.

      It is trivial, however, to read the *asserted* date, but as we say in my profession, don’t rely on it for any security related judgements. In the case of document provenance (author ,e.g.) that’s effectively a form of access control or the like and so would apply.

      Reply
  3. sahpa

    Agreed with the other commentator. It is extremely unlikely that Pangram’s success is due to its cheating by reading metadata.

    Reply
    1. Justin Fisher

      To be worth using, a detector needs not only (A) not get very many false positives, but also (B) get enough true positives to be worth using, and (C) continue to get enough true positives to be worth using even after it is widely used and the “market” is incentivized to find workarounds. It’s practically impossible to get all three of these. You can temporarily get A and B, but the market will soon find ways to evade it. Or you can get B and C, but only at cost of losing A.

      Reply

Designed with WordPress