News and views about philosophy, the academic profession, academic freedom, intellectual culture, and other topics. The world’s most popular philosophy blog, since 2003.
“given the actual number of career-worthy jobs (i.e. not per-course or adjunct positions)” The “id est” seems mistaken. Many tenure-track…
An interesting, and in my view, entirely welcome dimension of the “new” PPA has been a return to discussion of…
Daniel says a lot I agree with but on a few of the issues I’d slightly disagree. We’re facing the…
I doubt enough experts have read enough of the articles published in the top 10 value theory journals over the…
It’s hard to see the old PPA publishing some of what the new PPA has been publishing, although that’s not…
Richard, this is all fair and certainly, impressions of the current quality of the discipline are a subjective matter. I…
The stigmatisation of certain theories, counter-intuitive to many, as ‘preposterous’ should be avoided. It is not ultimately conducive to discovering…
This is interesting, although the "human" comparison class is not the strongest one. The humans still outperform AI, but AI is not that far behind. Curious to hear how informed readers (not random passersby!) interpret these results.
Notably, it looks like this experiment was done as a test of someone else's earlier experiment, who found that GPT3 did better than UCLA undergrads. It's interesting that in the new experiment, GPT3, GPT3.5, and GPT4 all do just about as well as GPT3 did in the earlier experiment, while the humans paid on the internet did much better than them, and the UCLA undergrads did a bit worse. "Better" and "worse" were evaluated by the researcher's judgment of which completion of an analogy was "correct" (e.g., abcd is to abce as pqr is to ???, where the researcher said that pqs would be correct, though of course there is a case to be made for various other answers – the summary linked here doesn't indicate how often either one of the GPTs or one of the humans gave an answer like pqe or pqrt vs something like gxsp).
Another notable fact is that the particular "analogical reasoning" task described here is an abstract one that is optimized for old-fashioned AI to be good at. (The author of this blog post says she did her PhD dissertation on designing a symbolic AI that solved this task well on the basis of rules.) In many ways, GPT-type AIs have done worse than humans at tasks that symbolic AI used to do much better than humans (for instance, multiplying 3 digit numbers) so it's interesting that it might be catching up to humans on this one task that older symbolic AIs had also done as well as humans.
More "realistic" analogical reasoning tasks that involve real-world knowledge are ones where old symbolic AIs were pretty hopeless, but it would be interesting how GPT responses compare to human ones. (And of course, analogical reasoning like this is something where the entire idea of a "correct" answer is at best tendentious, so it's hard to use something like this as a bench mark of "skill", but definitely still interesting to see the results.)
It is anticipated that models will dramatically improve in the coming months, enough so that it is premature to draw conclusions about what autoregressive large language models are capable of, before they are scaled to their capacity, and what shape the performance envelope will take once they are saturated.
LLMs are pretty remarkable. Humans see something on the order of 1 or 2 billion words in a lifetime, a tiny fraction of what current LLMs are trained on. But we have a visual system and a body to move around, modalities that LLMs currently do not have. Knowledge that humans only acquire through developing and interacting with the world, traces of that knowledge are present in language but only at a scale that can be “seen” by an LLM. Nobody knows how much of this knowledge is encoded in language, nor how far this strange new telescope can be refined to see more of what remains to be seen.
—–
KEYWORDS:
Primary Blog
Leave a Reply