Several law professors trained AI on the class textbook, and AI still gave incomplete or wrong answers half the time!

UPDATE:  A senior scholar of ancient Greek and Roman philosophy elsewhere writes:

The other day I gave the same query to two different AI's (anthropic.claude-3.5-sonnet.v2 and openai.gpt-4o):

"I'm interested in Plato's Republic, and I'd like to read some recent scholarly literature concerning his proposals for censorship in books 2 and 3 of the Republic. Can you suggest some articles written within the last ten years?"

One gave me two citations, one gave me five. Of those seven, exactly 1 was a real citation to a real article. The other 6 were mix-and-match nonsense: the names of real authors (lots of my friends!), the names of real journals, and titles that were plausible pastiches of title-elements.  But none of them actually exist. 

People tell me that AI is useful for some things, but it is still far from a serious adjunct to scholarship.