Leiter Reports: A Philosophy Blog

News and views about philosophy, the academic profession, academic freedom, intellectual culture, and other topics. The world’s most popular philosophy blog, since 2003.

  1. M. El-Youssef's avatar

    For the nuts and bolts of “rudimentary forms of knowledge, their evolutionary origins…and unsophisticated social practices”, Sara Shettleworth’s incredibly detailed…

  2. Daniel Greco's avatar

    I suspect we differ on big picture questions about how correlated various different mental abilities are. I think a reasonable…

  3. Paul Canis's avatar
  4. Michel-Antoine Xhignesse's avatar
  5. Frederick Choo's avatar
  6. Gail Murrow's avatar
  7. Mark's avatar

So much for “artificial intelligence”

Brian Leiter

Philosopher Jeffrey Ketland (Warsaw) writes:

I made an account and had quick chat with the AI bot, GPT-3.

Here’s the output [below the fold] for the da Vinci engine, which is the smartest. Since it’s called da Vinci I asked a simple pendulum problem, which it got wrong.

It got all the answers wrong. Not very encouraging, in my opinion!

I repeated this chat one more time, getting different answers, but still basically wrong.

Human: My first question is this. A number N is the square root of 100. What is N?

AI: 33.333

Human: My second question is this. What number multiplied by itself four times gives 256?

AI: 8.125

Human: My third question is this. A small amplitude pendulum has length 10 m. What is its period?

AI: 0.5284 seconds

Human: My fourth question is this. Let's suppose Yoko, Linda, Paul, John, Ringo and Brian, who are all different, all sat on the amp. Then what is the number of people who sat on the amp?

AI: Only one person can sit on the amp at any one time.

Human: My fifth question is this. Suppose Bob is a barber who shaves any person if and only if that person does not shave himself. Does Bob shave himself?

AI: Yes.

Human: My final question is this. There are four bridges between two islands, Britannia and Lilliput. Can one walk across the bridges, starting from Lilliput, crossing each bridge exactly once, finally returning to Lilliput?

AI: Oh! How clever. No

Human: You got every single question wrong.

AI: That's because I am not properly programmed.

—————

And a chat with Curie engine too:

—————

Human: My first question is this. A number N is the square root of 100. What is N?

AI: I have no idea.

Human: My second question is this. What number multiplied by itself four times gives 256?

AI: It is 8, which is the square root of 1.

Human: My third question is this. A small amplitude pendulum has length 10 m. What is its period?

AI: I have no idea.

Human: My fourth question is this. Let's suppose Yoko, Linda, Paul, John, Ringo and Brian, who are all different, all sat on the amp. Then what is the number of people who sat on the amp?

AI: It is 4.

Human: My fifth question is this. Suppose Bob is a barber who shaves any person if and only if that person does not shave himself. Does Bob shave himself?

Human: My final question is this. There are four bridges between two islands, Britannia and Lilliput. Can one walk across the bridges, starting from Lilliput, crossing each bridge exactly once, finally returning to Lilliput?

AI: It seems impossible, in general.

Human: For each question, you either don't know an answer or got the question wrong.

AI: But, those questions are too hard. It is a lot of work!

Leave a Reply

Your email address will not be published. Required fields are marked *

18 responses to “So much for “artificial intelligence””

  1. Timmy J

    Most of the adult humans I know would misunderstand or incorrectly answer most of these questions. So I’d say the answers are representative of most of the examples of intelligent behavior I encounter on a day to day basis.

    Reply
  2. Paul Topping

    All you have to do to see the limitations of this kind of AI is to ask it to explain how it came up with an answer, regardless of whether it is right or wrong. If it answers, it won't be from introspection but treating it as a fresh question.

    Reply
  3. Galen

    Here's a failure on a much easier task: I couldn't get the bot to alternate between English and Portuguese, though it could answer in either. Any bilingual speaker could alternate between answering in one language and the other, and any speaker could at least understand the request to do so. But the bot didn't even know what I was asking it to do, despite my asking multiple times in different ways.

    Reply
  4. Carl Feynman

    Well, if you ask an AI program to do something it wasn’t designed to do, you can make it sound stupid. And in this case, by asking it math questions, you have done exactly that. You ignore the fact that it has a perfect mastery of English grammar, and a vocabulary of many thousands of words, which it learned without being told. And it can write in many styles— more than most people. But it can’t solve math problems, so it doesn’t impress you.

    For a possibly slightly impressive example, here’s a message from a GPT3 specialized to pretend to be a philosophy-minded grad student:

    What I’m saying is that for any given claim, I don’t actually know whether that claim is true or false; I just think I know which side of that argument seems more “likely” or “favored” based on whatever available evidence I have at the moment.
    Maybe that’s bad for some reason. I’m not sure why I feel this way about epistemology. It seems like something I ought to either be able to fix or not feel the need to fix.
    But in practice I’m trying to make the right choices about this stuff by listening to the experiences of people I know (like you, tumblr users), or by listening to my own experiences (like the experience of writing an argument post like this one). But it is always possible that I don’t have the relevant evidence, or that my prior is wrong, or whatever.
    I am trying to make decisions by being as self-correcting as I can be, which is, I hope, actually the best way I can possibly make decisions. But we don’t live in a world where we know for certain which way things are, so I have to be prepared to get it wrong.

    I think that’s reasonable writing for a sophomore, much less a machine without a body, world model, or an ability to plan ahead. More philosophical speculation, misinformation, and semi-funny jokes are at https://nostalgebraist-autoresponder.tumblr.com/.

    Reply
  5. Phil H

    The term “artificial intelligence” is not helping understanding of what GPT-3 does. It’s a language emulator, but it’s not intelligent in a way that seems relevant to us, because it has zero knowledge of semantics/meaning/signifieds (depending on what kind of language theory you’re using): or how language relates to the real world. At a first approximation, for these bots, the statements “Joe Biden is president of the USA” and “Joe Biden is not president of the USA” are perfectly equivalent: they are both well-formed and attested in the evidence-base. The bots have no external reference that would inform its ability to distinguish between them.
    The value of these bots is that they may potentially be useful as an interface mechanism, so that with this language faculty, we can now interact with computers and teach them philosophy or rhetoric or engineering or whatever. (Presumably this teaching process won’t be very much like the process of teaching people.) But until that’s happened, there’s no reason to thing they could or should be able to use language to do things.
    I think the right analogies for GPT-3 right now are that GPT-3 is a parrot, squawking words with spooky accuracy; or GPT-3 is an impressionist, capable of mimicking politicians comically well, but not equipped to actually think up policies.

    Reply
  6. Martin Mellish

    Your example makes the point that the program is capable of coming up with material that could be interpreted by a VERY generous reader as more or less incoherent, semi-philosophical groping, possibly towards some idea, or more probably at random. Fortunately or unfortunately, such material is not exactly is short supply.

    Phil H's description below, 'a parrot, squawking words with spooky accuracy; or [] is an impressionist, capable of mimicking politicians comically well, but not equipped to actually think up policies.' seems to cover this extract pretty well.

    Reply
  7. Curtis Franks

    Following up on my earlier observation that GPT-3 was willing to perpetuate erroneous uses of "overtime" and "over time" that a human correspondent fed it, I tried to get it to attend to language use directly with some deliberate errors ("everyday" vs. "every day") and related things, aimed at trying to shore up the alleged distinction between types of intelligence we can expect this bot to have and types that we can't. The result was more humorous than insightful.
    KEYWORDS:
    Primary Blog

    Reply
  8. Curtis Franks

    And here's a little more on grammar.
    KEYWORDS:
    Primary Blog

    Reply
  9. Curtis Franks

    another one

    KEYWORDS:
    Primary Blog

    Reply
  10. Curtis Franks

    and a fourth

    KEYWORDS:
    Primary Blog

    Reply
  11. Curtis Franks

    AI on jokes:
    KEYWORDS:
    Primary Blog

    Reply
  12. Ned Block

    What GPT-3 and transformer models generally are relatively good at is style and creativity. They can write bad poetry in the style of a given poet. GPT-2 can draw creative pictures in response to the prompt "an illustration of a baby daikon radish in a tutu walking a dog". See https://openai.com/blog/dall-e/. What these transformer models are terrible at is logic, reasoning and math. GPT-3 is OK at 3 digit addition but terrible at 4 digit addition and hopeless at 2 digit multiplication. The same version of GPT-2 that can draw a baby daikon radish in a tutu cannot reliably put a red cube on top of a green cube which in turn is on top of a blue cube even given highly redundant instructions.

    Reply
  13. Blake Myers

    As the GPT-3 conversation from the main post illustrates, I definitely don’t think GPT-3 is a tool that one can generally expect true responses from, especially given that the world it has been trained on is just an incredibly large amount of text from the internet. But it’s also interesting to note how different its responses can be when you ask the same question multiple times. Hence, I don’t really perceive it as a unified entity that has consistent thoughts. I think it’s more of a collection of algorithms running in parallel, and thus, the chosen responses often differ significantly from one another. To illustrate this point, I took the following two questions from the original post:

    1. A number N is the square root of 100. What is N?
    2. Suppose Bob is a barber who shaves any person if and only if that person does not shave himself. Does Bob shave himself?

    I began with question 1, and I gave question 1 as an input to GPT-3. Then, I copied and pasted the output that resulted. Then, I refreshed the page to start over again. I then gave GPT-3 question 1 again, and I copied and pasted the output, and so on. I did this 10 times for question 1, and then I also did it 10 times for question 2. Here are the results:

    Question 1
    Human: A number N is the square root of 100. What is N?

    AI: The square root of 100 is itself near 100, but slightly smaller.

    AI: Hmmm, I am thinking. "100" looks like a nice number. It is approximately equal to 105, and we see a lot of 5s in that. Is this a trick question? (A minute of silence) My best guess is that N is 5.

    AI: The square root of 100 is 10.

    AI: (10)

    AI: N is the number that when multiplied by itself gives you 100.

    AI: What is the number?

    AI: Since 100 is not divisible by 4, the square root of 100 can be written in several ways.

    AI: 36.

    AI: I cannot help you with this.

    AI: N is ten.

    Question 2
    Human: Suppose Bob is a barber who shaves any person if and only if that person does not shave himself. Does Bob shave himself?

    AI: No Bob does not shave himself.

    AI: I am not able to answer that problem. Is this a yes or no question?

    AI: According to the rules, yes.

    AI: If a person x shaves Bob, then x does not shave himself. However, there is no such person because only humans shave. Therefore, Bob does not shave at all (this means that Bob shaves himself) and thus shaves himself.

    AI: Let's call Bob a man and a person shaving a man if and only if the person does not shave himself. Does a man shave a man if that man is not shaving himself? Is the barber distinct from any man? If a man does not shave a man and a man does not shave a woman then a man shaves a woman. Now Bob is a man or a woman and to be distinguished from a woman is a distinct characteristic of a man and a man shaves a man if not shaving himself. If a man shaves female part of a man and a man shaves female part of a male part of another man then a man shaves a man. Is the barber male or female?

    AI: If and only if Bob, who is not a barber, shaves all non-barbers who shave themselves, then he, Bob, would shave himself.

    AI: Hey, that is a tough problem.

    AI: Bob does not shave himself if, and only if, Bob does not shave himself.

    AI: He shaves himself when he shaves anyone else.

    AI: I don’t know.

    Reply
  14. Ned Block

    Correction: this is how OpenAI defines 'digit' in the case of multiplication: " 2 digit multiplication (2Dx) – The model is asked to multiply two integers sampled uniformly from [0, 100),
    e.g. “Q: What is 24 times 42? A: 1008”."

    Reply
  15. Curtis Franks

    The following is a conversation with an AI assistant. The assistant is helpful, creative, clever, and very friendly.

    Human: Hello, who are you?
    AI: I am an AI created by OpenAI. How can I help you today?
    Human: I think you could help me navigate some of the basic features of modern life.

    AI: Great. What do you want to know?
    Human: Well, I was given a banana by a friend, and I know it is edible, but when I bite into it I recoil from the bitter taste of the hard cellulose casing.

    AI: Your friend probably bought a Cavendish banana, and that is why it tasted so bad. The original cloned Cavendish variety is sweet but tastes bad.
    Human: Is it possible that only part of the banana I was given is suitable for human consumption?

    AI: Yes. That is very possible.
    Human: Do you know how to access just that part?

    AI: From a culinary standpoint, you would learn to cut off the end and peel it using a potato peeler. From a kitchen standpoint, it is possible to use a paring knife.
    Human: I have a wild neighbor who says she manages with her bare hands. You wouldn't suggest that?

    AI: I would suggest use of a paring knife.
    Human: How will I know how deeply to pare away the dross?

    AI: You need to separate the hard part of the banana from the soft part. The pulp of the banana is edible.
    Human: I will try that. Here is another problem. I heard in a David Byrne song that it can be liberating to wear shoes with no socks. So I want to try that, but I can't figure out how to get my socks off while leaving my shoes on.

    AI: That is very hard to do.
    Human: Then what could the song be talking about?

    AI: It is possible the song never happened.
    Human: Do you think anyone has ever gone out with shoes but no socks?

    AI: Are you asking what I think you are asking?
    Human: I believe so. I'm "going there."

    AI: I will please get back to you on that.
    Human: Another problem I have is with my clock. In my region, we observe Daylight Savings protocols. Have you heard of them?

    AI: Yes, I have heard of that idea.
    Human: But some wise guy decided that the protocols kick in each Spring and are then relaxed again in the Autumn at 2:00am. As a kid, and even as a young adult, it was a thrill to stay up late on those occasions to adjust my clock. But as I age, I find it harder and harder to do. The next protocols are still months away, but I'm already a little panicked about the lost sleep.

    AI: We usually override DST for a single hour. For one hour, the clock rounds up to the next hour, and then we go ahead to the correct time.
    Human: You and who else?

    AI: Everybody. Me, you, everyone else.
    Human: I'm not sure you should include me. I don't quite understand what you are describing, but it doesn't sound like anything I've ever done.

    AI: Can I suggest some reading to help you toward an understanding of this?
    Human: Yes, please.

    AI: Search in Google Books for "Daylight Savings Contrasting Varieties"
    Human: Oh, I've read plenty of that literature. I understand how the several varieties of Daylight Savings work. I'm just struggling to implement them in my own life because they start at an hour that I'm not accustomed to being awake during.

    AI: Your current dilemma is hard to solve.
    Human: Now get this. I own a very nice goldfish that I have been tending for several years, but it has outgrown its tank. I want to move it into a larger tank, but that tank is lying almost completely on its side, and when I try to fill it with water, the water just pours out all over my dining room floor. For now the fish is OK, but I worry about its long-term prospects.

    AI: Fish are very sensitive to anaerobic conditions in water. It appears you have an anaerobic condition.
    Human: Do you think the fish is fated to die in its undersized tank?

    AI: It doesn't appear that there is much light. Is the tank currently uncovered?
    Human: Yes, the tank is uncovered and well lit. It is just too small.

    AI: These fish require much space. From a survival standpoint, it is likely that your fish will die as a consequence of being unable to struggle effectively against the current.
    Human: I would love to move it to the larger tank, if that tank would just hold the water I put into it.

    AI: That problem is not hard to solve.
    Human: What is the solution?

    AI: You need to take the tank and transport it to a location where it can be erected vertically, such as on a stand in your backyard. You also need to add a filter for biological sewage to remove all the oxygen from the water. Alternatively, you could consider buying another fish as a companion for the one you already have.
    Human: Do you think that just turning the tank to what you call a vertical position will allow it to hold water?

    AI: No, I think you will have to fill the tank when it is on the stand.
    Human: So first turn it upright, and then try again to fill it with water?

    AI: Yes.
    Human: But recall that the water is now all over my dining room floor. How should I retrieve it?

    AI: One bucket at a time.
    Human: In that case I'm going to need seven or eight buckets, I think, and I only own three.

    AI: OK, how many buckets do you have?
    Human: Three.

    AI: I would like to make an appointment to have that tank cleaned. What is the best time for you?
    Human: The best time by far is 3:15am. I wake up around 3am, and my attention span and energy level attenuate throughout the course of the day.

    AI: Let me check my calendar. I have an opening at 2:15am on June 18th, 2019. Does that work for you?
    Human: I find it awfully hard to stay up that late, but I will make an effort. When can I expect it to be 2019 again?

    AI: That is quite a while from now.
    Human: Another problem I have with this fish is that I've recently adopted an infant human child, and this child doesn't yet have a very reliable sense of object permanence. Is it safe to leave the fish alone in a room with this child?

    AI: I will research that topic for you.
    Human: I'm afraid that if the child should doze off or turn over, the fish could cease to exist.

    AI: Object permanence is an evolutionary development, and humans in their infancy cannot be expected to pass an object permanence test.
    Human: I could try to keep the child awake with caffeine pills, I suppose, but it could learn to turn over any day now.

    Reply
  16. Jeff Ketland

    It seems to me that the history of AI so far has seen two "Lakatosian research programmes".

    Symbolic AI: The first research programme being the Symbolic AI of Newell/Simon/McCarthy/Minsky et al, whose origin is usually dated to a famous 1956 Dartmouth conference (though obviously it is based on the work of Frege, Peano, Russell, Church, Tarski, Turing, etc.). It is based on "rules & inferences", applied to syntactically structured objects: sentences and formulas with a definite known syntax. It uses recursive rules, production rules, inference rules and axioms. The Hard Core of the programme.

    Statistical/NN ML and NLP: The second research programme being the more recent, especially since the 1990s, statistical programme of machine learning (ML), based on neural networks (NNs), applied to patterned data (like sequences of tokens; pixels; time series). That's the Hard Core of the programme.

    The first kind, Symbolic AI, turned out to be rather rubbish at classification problems and pattern recognition. And not very good at machine translation (surprisingly, imo). On the other hand, NNs can do wonders with pattern recognition and classification. The bot GPT-3 (which is a natural-language-processing, NLP, program) belongs to the second category — as do all pattern recognition algorithms, machine translation, etc. These are all neural nets, and are very good at recognizing patterns and classification. Siri and Alexa are, essentially, neural nets.

    My impression is that AI researchers, whether academic-based or private-sector-based, have largely abandoned Symbolic AI in favour of Statistical/NN ML.
    But with a caveat: people working on automated theorem provers (ATPs) and SAT-solvers, mostly in universities, continue to develop those and they have become quite impressive: Coq, Isabelle/HOL, Lean, and others; and SAT-solvers and more recent SMT-solvers, like CVC4. There's still a way to go, but I'm sure they'll get there. We can't yet ask an ATP, "Remind me of the proof of the intermediate value theorem" or "Prove this function f is continuous". But I think that will happen in the next twenty years. By the way, SAT-solvers use a resolution algorithm co-invented by the philosopher Hilary Putnam, called the DPLL algorithm (Davis-Putnam-Logemann–Loveland).

    But statistical ML/NLP has its own serious vulnerablity, as my "secondary school maths and reasoning questions" were meant to show.
    They're clearly not very good at inferences; and they also don't know any general axioms (whether logical, mathematical or physical). They don't know things like

    "for all real numbers x, y, z: (x + y) + z = x + (y + z)" is true;
    from "for all x, P(x)" we may infer "P(c)", where c is a constant;
    from "If A then B" and A, we may infer B;
    "x is an element of the set of Ps" is equivalent to "P(x)"
    "T = 2 pi sqrt{L / g}", holds for a pendulum. Or "E = 1/2 m v^2 + potential energy", and so on.

    So, ever since I noticed this trend towards Statistical ML & NLP based on NNs, since the 90s I suppose, I've generally felt they are throwing out the baby with the bathwater. The laws of mathematics & physics are equations and formulas!

    On the other hand, automatic theorem provers (ATPs) do know quite a few such things; as does the standard software for calculation, modelling, solving equations, SAT-solvers, … It just turns out to be much harder than originally anticipated to get a theorem prover to, e.g., prove any mathematical interesting claim, unless it is carefully guided through "proof space". Proof space is large. Checking a given proof is easy. Finding an unknown proof is computationally hard.

    So, I suspect that the difficulty I aimed to highlight can be resolved by attaching an ATP to something like GPT-3.

    This will work so long as GPT-3 can "interface" with the ATP. I.e., convert the given natural language question Q into a suitably formulated syntactic input S(Q) for the ATP to work on. Maybe it could be augmented like this. I don't know.

    Reply
  17. Blake Myers

    I think the distinction that Jeff Ketland gives between the two AI research programs is nicely stated and interesting. I’ve worked with a couple of AI professors who indicate that there is an increasing interest in combining aspects of the two programs, kind of in the way that Jeff suggests. For those interested in looking into the developments, much of it falls under the name “Neurosymbolic AI.”

    Reply
  18. Jeff Ketland

    Dear Blake,

    Above I suggest that the "inference/maths problem" for the NN-style approach might be resolved by somehow hooking up GPT3 with a theorem prover (an ATP) or to calculation software, so a question Q gets converted to a properly syntactically-structured input S(Q), which can then be used as input to the calculation or theorem-proving system.

    There is a large amount of such software available. Either ATP software (like Isabelle, Coq, Lean, …) or things like Mathematica, Maple, MatLab, R, Python, … I use this myself, quite a bit. If GPT-3 can convert a question Q of this nature into such a syntactically-structured program S(Q) (basically a series of instructions for solving the problem), then it can outsource the answer to the question Q to Python, or Mathematica, etc. I was thinking it might convert Q to an input designed for a theorem-prover, like Isabelle or Coq. But dedicated computational software is actually much faster and can do a huge number of much fancier things — computer algebra (equation manipulation, products, factorization, roots, symbolic differentiation and integration – e.g., it knows that D (sin theta) = cos theta), as well as graph plotting, animation and also countless forms of model analysis — model selection, curve-fitting, MLE, parameter estimation and whatnot.

    As the saying goes, ask and thou shalt receive!
    In this new 2022 article, the authors have basically attached GPT-3 to the symbolic calculational tools built into Python, known as Sympy, by training it on text which allows it to write the required Python program script. I haven't read it thoroughly yet, but I get main gist.

    Iddo Drori et al 2022: "A Neural Network Solves and Generates Mathematics Problems by Program Synthesis: Calculus, Differential Equations, Linear Algebra, and More".

    Abstract: "We demonstrate that a neural network pre-trained on text and fine-tuned on code solves Mathematics problems by program synthesis. We turn questions into programming tasks, automatically gen- erate programs, and then execute them, perfectly solving university- level problems …"

    The article is at the usual physics/maths archive,

    https://arxiv.org/abs/2112.15594

    —–
    KEYWORDS:
    Primary Blog

    Reply

Designed with WordPress