Listen now (60 mins) | 0:00 Gary’s background and place in the AI world 1:17 Are Gary’s views on AI paradoxical? 8:48 Bob: Searle’s Chinese Room argument is dead 19:01 Have LLMs demonstrated "theory of mind"? 26:06 Arguing the semantics of an LLM's “semantic space” 31:36 Do LLM representations map onto the real world? 40:52 Can (and should) we slow down AI development? 51:43 Gary: I’ve never seen a field as myopic as AI today 54:25 The (symbolic?) future of AI Robert Wright (Bloggingheads.tv, The Evolution of God, Nonzero, Why Buddhism Is True) and Gary Marcus (https://garymarcus.substack.com/, Humans vs. Machines, The Algebraic Mind, Kluge). Recorded February 13, 2024. Twitter: https://twitter.com/NonzeroPods
In my listening experience, when a podcast guest describes a tense interview or conversation as "a lot of fun" (or some derivation thereof) it means they are never coming back. I hope that isn't the case here, because I really enjoyed this episode. Civil, mutually respectful, disagreement.... it's a lost art, like silk weaving or coppersmithing.
Article suggestion: a synthesis or comparison of Gary and Nora Belrose's take on whether LLM's can "understand" or have "real" semantics. You also have good material from some of these comments here, particularly the ones that are pushing back on your claim that LLM's have understanding or, in the case of your schadenfreude example, have some kind of cognitive empathy. It'd be a good way to clarify the different positions, which I image you'd have to do anyway for the AI book you're writing. Wright or wrong, both the Gary and Nora interviews are gold.
Pointing out ineptitudes of current AI is interesting, so far as it goes, but does little to address doomer concerns. It could be seen as a perpetual act of goalpost-shifting, but definitionally, if it isn’t the AI that causes our destruction, it wasn’t the one we were talking about.
No one thinks GPT-4 is that AI. But many were right to think ‘holy shit’.
It’s not too comforting to know human extinction is difficult; starting over with stones as pillows is better than nothing but it doesn’t fill me with as much optimism as does the starship Enterprise.
What’s required is a little imagination. A little David Deutsch. What is possible with intelligence? What is the limit of intelligence?
How alien it will seem. How unknowable.
It’s never too early to start grappling with the implications. After all, which seems closer, a powerful machine intelligence or effective global governance that might mediate something of similar impact?
This was an extremely interesting conversation which goes to the heart of a good deal of the internet hype around AI. I listened to it twice. Gary has a deep and nuanced understanding of AI, what it is, what it can do and what it can’t do. Below Bob asks both JJ and me what we think he got “literally” wrong in his talk with Gary Marcus.
The first point that Bob has literally wrong, is his argument that LLMs have an “understanding” of what they spew out. LLMs have nothing that could reasonably called “understanding” of anything that they say or do. As Gary points out repeatedly, an LLM does not know anything about how the world is or operates. It has no judgment, motivation, goals or understanding of the world. What it does, is spew out information based on statistical correlations among words and word usage based on the materials it is trained on. That’s why it delivers so much false, “hallucinatory” and other absurdly wrong information. An LLM has no way to anchor itself in reality, it has no subjective perspective, goals or motivation and is indifferent to what it says and to what you ask it to do.
So Bob is incorrect both on his suggestion that LLMs might have “understanding” (or any kind of sentience, consciousness or subjective experience). LLMs can do what Siri does, such as look up movie times at your local theater but that doesn’t mean it understands what a movie or what a theater is. It is looking information up on the internet as people used to look up words in dictionaries.
This also means that LLM are far less of a threat than Bob thinks. The biggest danger of LLMs is the the fact that people may rely on them, as Air Canada recently discovered to its horror when it was held liable for a non existent discount program that its generative AI made up and convinced customers to use. Gary’s point about self driving cars is also key here. While self driving cars can do amazing things, they don’t work and aren’t reliable and cannot be made reliable because their understanding of the world is too limited to avoid monumentally stupid decisions like driving into an airplane because they don’t know what an airplane is and don’t expect to see it on the road.
The conversation about syntax and semantics was itself really just a semantic argument. It didn’t inform or resolve any issues. What Gary really meant though is that LLMs don’t understand what words mean, they just look at statistical correlations of the occurrence of words. Characterizing what LLMs do as semantic or syntactical is just a not very useful characterization. The real point is that LLMs aren’t sentient and have no what they are doing in relation to the external world. John Searle’s Chinese room experiment isn’t even worth talking about here because it is a lousy thought experiment and it isn’t at all how LLMs work.
Bob is also wrong on the idea of the possibility of implementing an AI slowdown or pause. The whole world is working on accelerating the development of AI and nothing can stop that. The stock market is driving investment of billions in AI research which is the hottest field in business today.
Having the US put a tax on AI would simply cause AI research to move outside the US. And there is no possibility of getting the whole world to adopt such a tax. And if the whole world did in fact adopt a global tax then the big companies that can afford it would simply pay the tax and keep going as they do with every regulatory addition. So the effect would just be to push even more of the development into the hands of big companies that can afford any tax imposed.
So Bob, those are some of the things you were “literally wrong” about in this conversation and why.
So are you defining 'understanding' as entailing subjective experience? If so, I don't think that's a useful definition, since we have no way of knowing whether AIs have subjective experience. And if not, what is your definition of 'understanding'? In other words, what properties would an AI have to have in order to have understanding by your definition? (Also, do I take you to be saying that I've said LLMs have subjective experience? I've never said that and am agnostic on that question.)
Yes the word "understanding" does entail awareness and that involves subjective experience of some kind. If "understanding" doesn't involve "awareness" those terms lose all meaning. And one thing that is very clear is that computing statistical correlations does not in any sense involve "understanding".
So we do absolutely know that LLMs do not have that kind of subjective
awareness or sentience or consciousness or understanding (or whatever you want to call those qualities of mind) since all LLMs do is the computation of statistical correlations.
LLMs can determine that the word "love" is oven used near the words "I" or "you," but they have absolutely no idea what the word "love" actually means.
I would strongly suggest you read Philip Ball's "The Book of Minds" which is available on audible and thoroughly discusses all of these issues and the history of AI, including the move from symbolic to
The great chess and Go pjaying computers like Deep Mind and Alpha Go don't even know they are playing a game and don't care if they win. They are just doing computations. The key point is that you cannot understand meaning if you don't have any goals or motivations. LLMs and all existing AIs lack goals or desires and so they don't understand meaning.
If you read Ball's book you will give up your "semantic" argument because it jut makes no sense in light of what LLMs actually do.
And that applies to your schadenfreude example. As Gary points out in your talk, there is no reason to think that isn't just another statistical correlation. If you are embarrassing someone publicly, normally some people in the audience like that embarrassment. It's not at all a rare occurrence.
Thanks for the clarification. I don't agree that we "absolutely know" that LLMs don't have subjective experience. I think there's no way (for the time being at least) for us to know whether they do or don't have it. And that's why, if you're going to define understanding has inherently involving subjective experience, I don't see any point in debating whether AIs have understanding. I elaborate on this in a piece about Searle's Chinese Room thought experiment that I'll post this week. (BTW, I'm not saying it's *wrong* to define understanding as entailing sentience. I'm just saying that if you want to have a useful debate about whether AIs have understanding, you need to choose another definition.)
Thanks much for your reply. I am actually making a broader comment.
Subjectivity is fundamental to understanding. I don’t think you can redefine “understanding” so that it can exist separate from awareness and subjective experience. Otherwise the word ‘understanding’ isn’t doing anything in your argument. You cannot have meaning without subjective experience because there is no context for it and meanings are always contextual, not objective. For an information theory approach to the same issue see Christoph Adami’s excellent conversation with Sean Carroll released just this morning.
On the point that "[o]therwise the word 'understanding isn't doing anything in your argument," let me pose a related question to Bob: have you identified any example of AI behavior in which statistical pattern recognition/application alone would not explain the outcome--thus revealing that "something else" is happening in these systems? As a number of us have noted, your schadenfreude example is not actually an example of that. Do you have any others--i.e., examples of AI behavior that reveal something more than mechanical pattern recognition/application? If so: (a) what are they, (b) why do you believe they show something more than pattern recognition/application, and (c) and what is your explanation for those examples, at a nuts-and-bolts level? Unless you can provide such an example to illustrate your position, it's quite unclear what you are referring to vis-a-vis "understanding."
I have the greatest admiration for you as a public thinker. On topic after topic over the years, I have been so impressed by your perspectives and positions, and have almost always agreed with you.
Until generative AI. As I wrote in some comments I posted in the "Parrot Room" last year, many of your positions on generative AI simply don't reflect how the technology actually works. Marcus did a terrific job explaining some of the key errors in your positions. I had tried (obviously unsuccessfully) to make some of the same points in my posts. For example, I discussed the same "shadenfreude" episode, and like Marcus tried to explain why it didn't mean what you think it means.
I really wish you would step back and reappraise your positions on generative AI. I have the sense that you jumped to some conclusions early on, and ever since have been looking for reasons to defend those early positions. It's of course hard to admit error, but it's worse to stubbornly stick with positions that are not correct.
To be clear: much of what you say about AI is indeed correct--its massive power, its significant potential downsides, the need for great care in its development and deployment. But you are continuing to stick with a conception of how it works--at the mechanical level--that is simply not correct, and that is detracting from your larger commentaries on AI.
I would love to hear how you would grapple with the many fascinating implications of how generative AI actually works. That would be worthy of the book that you are writing--there are so many fascinating issues, ranging from policy questions to philosophical questions, that generative AI presents. But to truly confront those questions would require recognizing how generative AI actually works.
After generative AI became a big topic last year, you quickly developed a position against the "supercharged auto-complete/stochastic parrot" explanation of generative AI. You criticized that explanation as missing something deeper going on with generative AI.
I think you developed that position because you viewed the auto-complete/stochastic parrot explanation as somehow denigrating the power of AI and ignoring its potential consequences. But whatever your motivation, you took that position early, and have since dug in deeper and deeper.
The position is simply not correct. Let me make a few points (versions of which I made in the Parrot Room comments last year):
First, generative AI is fundamentally about statistical pattern recognition (in data sets) and pattern application/reconstruction (in response to user prompts)--just like autocomplete. The power of generative AI systems dwarfs what we had known as simple autocomplete, but the basic approach of identifying statistical relationships among characters in data sets, and then applying those patterns to user prompts, is the same. Terminology like "vectors" and "neural networks" can sometimes confuse this, because the words may suggest something else is going on. But if you strip away the jargon, and examine the actual mechanics, both generative AI and autocomplete are forms of pattern recognition/pattern application. Gary was correct to make the autocomplete comparison in your discussion with him.
Second, this is profoundly different than systems that reason from first principles, as Gary pointed out using a variety of examples. The generative AI systems are simply parroting language based on statistical relationships, not engaging with the underlying ideas represented by that language. That is why the "stochastic parrot" characterization is also correct.
Here is another example: imagine a society that has a well-developed understanding of math, and expresses their correct understanding in various writings about math. But, for whatever reason (this is a thought experiment), the society has a glitch--every member in that society believes that 2 + 2 = 5, and every member in that society reflects that error in their writings. This society would create a set of writings that (a) accurately describe abstract mathematical principles but (b) reflect the society-wide glitch in believing that 2 + 2 = 5.
If a generative AI system trained itself on the writings of that society, it would be able to recite math principles correctly--because the writings of the society properly described the abstract principles. But if you asked that AI system "what is 2 + 2?", the response would be "5." That is because generative AI is simply parroting the results of the patterns that it has detected, and in this thought experiment, the uniform, society-wide pattern was to say 2 + 2 = 5. The fact that this AI could also recite the abstract mathematical principles correctly does not mean it is "learning" those principles in the way that we do--as Gary explained--and does not mean it could use those principles to correct the society-wide error in believing 2 + 2 = 5. It couldn't, and it wouldn't.
Now, we use words to express our ideas, and our use of words create linguistic patterns, such that AI "parroting" the words in those patterns may give the appearance that the AI understands the ideas underlying the words. As you have noted, there are amazing examples of AI's generating texts that seems consistent with our human understanding of the meaning of the ideas underlying the words. But again, that is just a function of the statistical relationships among the words in the data set. Put another way, it is no surprise that AI systems will sometimes use words in patterns that seem consistent with our understanding of the ideas underlying the words--after all, the AI systems train on data sets that reflect humans expressing their ideas in words, which create patterns of language consistent with those ideas. But as Gary pointed out through several examples, parroting language that sometimes seems consistent with our understanding of the underlying ideas is very different than reasoning based on the underlying ideas--by, for example, applying those underlying ideas to contexts outside the original data set. (See his example of the Tesla running into the jet.)
I have the sense that your schadenfreude episode (which you have cited repeatedly) played a significant role in leading you astray on how generative AI works. You seem to have interpreted that as a remarkable example of AI understanding the subtle concept of schadenfreude, and applying it to a new setting (the one you described in your prompts). That's not correct. What was actually happening is that the AI system had identified subtle linguistic patterns in its training data involving the word "schadenfreude," and identified elements of those patterns in your prompt, such that it listed schadenfreude as a possibility. The AI doesn't actually know what schadenfreude means. You mistook its (incredible) power of linguistic pattern recognition/application to be an understanding of the underlying meaning of the word. You were wrong about that, and you've made similar errors in much of your commentary about AI.
Third, you sometimes (including in your conversation with Gary) say "well, we don't really know what is going on inside these systems," which I take to be a suggestion that there is something deeper occurring than pattern recognition/reconstruction. But we *do* know what is going on in these systems, in terms of the basic mechanics. When people talk about the "unknowns," what they are referring to is that the AI systems are picking up on subtle and nuanced linguistic patterns that (at least initially) are not discernible to humans. That is truly remarkable, and part of why generative AI is an amazing technology--the machine learning on enormous data sets yields pattern recognition that would have been unimaginable in the past, and that (as Gary said) is hard for humans to comprehend now. But that does not mean the basic mechanics are unknown--they are completely known. Put another way, we know how the machines are designed and how they operate--but the statistical patterns generated by those operations can be hard to comprehend. Yet even though they can be hard to comprehend, they are just statistical patterns among text characters.
Fourth, as I said above, saying that generative AI is "just" statistical pattern recongition and pattern application/reconstruction is, in no way, to denigrate the power of the technology. It is immensely powerful, and I share many of your concerns--e.g., the risk of bad actors using AI to proliferate disinformation. But it is powerful in a somewhat different way than you have suggested, and recognizing what it actually is has implications for the policy and even philosophical issues posed by AI.
Fifth, I encourage you to continue having conversations with folks who have similar views as Gary. For example, I'd love to hear you have a discussion with Emily Bender, one of the authors of the article "On the Dangers of Stochastic Parrots."
Finally: there are so many truly fascinating questions presented by AI, ranging from policy issues to deep philosophical questions. I tried to list some of those in one of my Parrot Room posts. I can think of no one that I'd rather see grapple with those questions than you. But you're just not doing it at the moment, because you set off early on a path that misconceives what generative AI is, and without a clear understanding of what it is, you can't properly engage with the true policy/philosophical implications of AI.
You make no mention of the fact that LLMs create a massive multidimensional 'map' of words in a kind of 'semantic space.' To me, that's the first thing to take into account if you're going to have a serious discussion about whether, and in what sense, LLMs 'understand' things. And to you it's not even worth mentioning? In any event, I emphasized that fact in my conversation with Gary, so if you're going to do a serious critique of my position, I don't see how you can ignore it. You keep repeating that the LLMs just sense 'statistical patterns' without taking into account the internal structures that the models build in order to be able to do a good of handling the statistical patterns. And again: We *don't* know everything about the internal structures yet. But we do know about the semantic mapping. If you don't think the semantic mapping is even worth mentioning, you and I see this whole thing so differently that I doubt further discussion of this would be productive. But I do appreciate your taking the trouble give me the feedback.
Bob, the "map" is comprised of the patterns of statistical relationships among the elements in the data set. It is indeed remarkable--that is precisely what I have said (repeatedly) in noting the immense power of LLM's to discern subtle and nuanced patterns in the data set. But the map is not more than those statistical patterns. Using the term "semantic space" is just deploying jargon, and adds nothing to the substantive discussion of the mechanics of these systems.
I've tried several times, including in my Parrot Room comments, to explain what I see as the problems in your perspective on generative AI. I either am not articulating these problems well enough, or you simply don't agree that they are problems. Either way, I'm going to throw in the towel on this discussion.
My last comment is I meant what I said: I can think of no one that I'd rather see grapple with the policy and philosophical implications of how AI actually works, than you. But to do that, you're going to need to admit some mistakes and move on. Nothing wrong with that! These are complicated issues, and having some early misunderstandings is no big deal. But sticking with those misunderstandings is going to continue to detract from your commentaries on AI.
I fully understand the sense in which what I call a mapping of words onto semantic space is based on statistical patterns in language. What I don't understand is why you object to the term 'mapping of words onto semantic space.' After all, those statistical patterns are to some extent reflective of the human understanding of the meaning of the words in the first place; so why should it not be possible to extract something from those patterns that warrants the term 'semantic'? Dismissing the term 'semantic' as 'jargon' is just name calling. You need to make an actual argument as to why the term doesn't apply.
My objection is that you seem to be using the term to suggest the "map" entails--at the mechanical level of how these systems work--more than the detection of statistical patterns in language. It doesn't. In terms of the mechanics of these systems, they are just using statistical patterns in language. That's it.
You are certainly correct that these patterns have some correlation to what humans perceive as the underlying meaning of those words, such that the AI's use of those patterns will give the appearance that the systems understand the meaning of the words (at least sometimes). In one of my earlier posts, I had a paragraph on exactly this. (It starts with this sentence: "Now, we use words to express our ideas, and our use of words create linguistic patterns, such that AI "parroting" the words in those patterns may give the appearance that the AI understands the ideas underlying the words.")
But this correlation does not mean the systems actually understand the underlying meaning of the words--they don't. You asked Ted Farris whether this question of "understanding" turns on whether the AI systems have subjective experience. I think subjective experience is part of it--I think subjective experience is an important part of how we engage with the underlying meaning of words. But as you note, we cannot prove the AI systems lack subjective experience. But I think proving the AI systems do not have "understanding" of underlying meaning does not require reliance on our intuitions about whether they have subjective experience. It can be proven empirically, including through real-world examples like the ones that Gary provided in your discussion, in which the AI systems do things that they would not do if they had command of the underlying meaning of the words they are using. And, I think my "2 + 2 = 5" thought experiment in one of my earlier posts makes the same point.
I also agree with Ted Farris that ultimately, whether we call the "map" a "map of semantic space" is itself a semantic issue. That is what I meant when I said deploying jargon doesn't add anything to the substantive discussion of the mechanics of these systems. One of the things that I've always loved about your writing and your discussions is that you have a gift for explaining complicated technology (e.g., quantum mechanics) in simple, plain English terms. The reason I've appreciated that approach is because it strips things down to the core issues, and allows those issues to be discussed without the distractions of constantly defining terms, etc.
Here, we know--and you acknowledge--the AI systems are detecting statistical patterns among characters in data sets. That's it, as far as I'm concerned--there is the detection of those patterns, and then the application/reconstruction of the patterns in response to user prompts. (As I have said over and over, "that's it" doesn't mean these are anything other than wondrously powerful---they are indeed wondrously powerful.)
If you think something else is going on beyond that, what is it? And how would you explain that without resorting to jargon, and without saying "we just don't know what is going on in these machines"? We *do know* the systems are developing and applying statistical models. If you think--or even just have a hunch--that there is something more than machines doing pattern recognition and pattern application, please explain what it is, in clear terms.
My principal problem with your approach to this subject is that, starting last year, you seem to have had an initial reaction that these systems must be doing more than pattern detection / pattern application--and ever since then, you seem to be looking for ways to defend that initial reaction. It's a fruitless quest: they are not doing more than pattern detection / pattern reconstruction. And I think if you try to take your best shot at answering what "what else are they doing beyond that?" in plain English, this would actually help show the problems with your position, by exposing the core issues stripped of jargon.
As I've said over and over, there are so many fascinating questions that remain, even if you were to accept that these systems are "just" doing linguistic pattern recognition / pattern application, without command of the underlying meaning of words. But you can't truly engage with those questions without accepting the reality of how these systems work.
Very well said JJ. I will comment further in due course. Gary's objections to many of Bob's positions were very well and precisely phrased and show a nuanced judgment that Bob can't seem to hear. Bob is sticking to positions he has had since ChatGPT4 was originally introduced which are both wrong and result from cognitive biases toward seeing an apocalypse, disaster or existential threat in most every significant technological or political development and seeing global governance (always impractical) or a stop to irresistible market or political forces as the only solution.
I've got an answer for you from your most recent 'Earthling Unplugged': "The whole thing about AI is it can in principle, as it evolves, do everything people do with their heads."
LLMs can do some things we do 'with our heads,' like manipulate language, get right answers to questions, notice and produce patterns, etc. But there is a huge class of things we do with our heads, to me the most important things, that it is *in principle* not capable of doing: it can neither understand nor express meaning.
[I know you are very impressed by what might be called a 'semantic map' found in LLMs -- that dog is mapped closer to cat than to aircraft carrier, etc. But nothing about the ability to reproduce accurate semantic patterns of this sort from huge swaths of text suggests actual semantic engagement (actual understanding/expression of meaning).]
None of this is to say we should stop worrying about the potential enormous harms of AI - bad actor stuff, job replacement stuff, etc. But I do think that we should START worrying about the discourse around AI collapsing or deflating how we view our own human capacities for understanding and expressing meanings.
Case in point, from your comments:
"LLMs help us see that language, rather than meaning something about "the world", works as a vehicle that moves the mind through meaning space. Inherently therefore, the meaning of an utterance depends on the starting point of the listener. Once again, this is obviously true; you have to believe in unempirical academic theories to believe that eg this paragraph has the same meaning to an informed reader and a Chinese child."
This is a view of meaning that is strikingly devoid of human agency. If what something means is simply 'what-it-is-taken-to-mean,' misunderstanding (and thus, understanding) is impossible.
I listen to a large number of podcasts across a wide spectrum of topics. I dislike the amateur podcaster mode of interviewing of talking way too much in the pretence of conversation. In this case Robin Wright sounded like a petulant child and I stopped halfway through despite being very interested in the subject and interviewee. I long daily for better interviewers to emerge but in this era of unbridled expressionism I fear this is another skill lost in the digital sands
Great discussion into an amazing and frightening world. All artificial intelligence should be subject to strict regulations by none corrupt politicians and Ai researchers.
My take is that this "pattern-matching" vs "understanding" thing is a false dichotomy. They are the points on a spectrum, and when we say "pattern-matching" we really mean the same thing as "shallow understanding".
Marcus claims that AIs fall apart when you step "outside the bounds" of the training data. But strictly speaking - almost everything is "outside the bounds" of the training data. Most sentences, including the one you are reading right now, have never been written down before. LLMs have clearly demonstrated an ability to operate outside "the bounds" - in that they respond in a reasonable way to many questions they have never seen before. The real question is "Can they reach human-level out-of-boundsness using current approaches - or is something fundamentally different needed"? THAT is where the debate could potentially be productive.
The examples Marcus gave are kind of questionable. The ones I've seen so far are:
GPT knows who Tom Cruise's mother is (Mary Lee Pfeiffer), but not (in an independent conversation) who Mary Lee Pfeiffer's son is (have confirmed - it's legit).
- This seems to just say that the model can contain knowledge that it cannot recall - humans do do this all the time - forget facts and are able to recall them later when queued correctly. I don't see how it indicates a lack of "understanding" on the AI's part - just that knowledge storage is asymmetric.
Hallucinations about Gary's pet chicken / Elon's car crash, etc.
- These are a little more interesting - but in the examples, the model was just prompted to say something - not told that it needed to be true. There is something interesting their about the model's inability to understand when it doesn't know something - but humans have similar tendencies (the term is "bullshitting"), albeit usually less extreme. Still - this doesn't seem to indicate a categorical difference in the way these things process information.
Math: inability to infer and execute algorithms.
- It's probably true - you can train these things on adding 10-digit numbers all day and they'll probably never get it right, despite begin able to recall the algorithm if prompted! (actually no longer true - the new GPT can write and execute code, and will correctly say that "The sum of 9576105326 and 3827630678 is 13,403,736,004"). But I'd argue the way the model responds (pre-code executor) is more or less how a human would respond if you asked them to calculate the result without pen or paper and they had to answer on the spot. GPT4 (like most humans) does not have an internal notepad to write down intermediate results and perform recursive algorithms on. But recursive thinking is not out of the realm of AI (just look at AlphaGo) so maybe the marriage of these approaches is the ticket.
In short - all these "counter examples" are things that humans do to some degree - none of them seem to me to indicate a fundamentally different kind of thing going on under the hood.
edit: I forget the "Draw me a room with no elephants in it". This one is actually more interesting - you still do get rooms with elephants. It seems to be more about a deficiency in DALL-E (the image-generation model) than GPT. It seems that the image-generation model doesn't really understand language - just associations of words and imagery coming from the captioned images it's trained on. GPT itself seems to have no problem with "Write me a limerick with no elephants in it".
Good discussion. This is the most subtle I've heard Gary Marcus be. Really liked his comparison to GPS systems. Re the intentionality/aboutness issue, I'd love for Bob to talk to an eliminativist like Alex Rosenberg to hear what he's thinking about AI and what it means for intentionality. Also, Raphaël Millière would be another great guest to talk about AI and philosophy. He was superb on Sean Carroll's Mindscape podcast and Bob could have a fruitful discussion with him.
Unlike most of your other commenters, I'm not an expert in how AI works; I'm just a lowly psychologist who "understands" a little bit about the still mysterious human mind. Given that AI is basically a highly sophisticated and sensitive correlation detector and nothing more, it seems unlikely to me that they "understand," at least in the same manner humans do, including ascribing significance. To think that they do reminds me a bit of the mistake some people make in thinking correlation = causation. But just to be sure, I went straight to the horse's mouth and asked ChatGPT, and here's some of our conversation, edited for brevity: Me: Do you possess understanding? ChatGPT: I don't possess understanding in the way humans do. I don't comprehend the information or have consciousness. My responses are generated based on patterns learned from data, and I don't truly understand the meaning of the information. I don't have awareness, beliefs, or subjective experiences. If my responses seem like understanding, it's a result of patterns and associations present in the data on which I was trained. Me: Can you think critically? ChatGPT: No, I don't have the ability to think critically. While I can provide information, analyze patterns in data, and generate text based on learned information, I lack true understanding, consciousness, and the ability to engage in critical thinking. Critical thinking involves evaluating information, considering different perspectives, and making reasoned judgments, which goes beyond the capabilities of current AI models like mine. Me: Do you possess metacognition? (Note: I don't think you can possess meaningful understanding without metacognition, and it's why ChatGPT sometimes makes very stupid mistakes). ChatGPT: No, I don't possess metacognition. Metacognition involves the ability to reflect on one's own thoughts and cognitive processes. While I can generate responses based on patterns learned from a diverse range of data, I do not have self-awareness, introspection, or the ability to think about my own thinking. Me: Why do people anthropomorphize you? ChatGPT gave a lengthy answer that described four major reasons why humans might anthropomorphize ChatGPT, including this: Some users might not be fully aware of the limitations of artificial intelligence models like mine. They may project human characteristics onto me due to a limited understanding of the underlying technology.
So there you go! When it comes to semantics, it seemed like you possess a somewhat simplistic representation of a very complex topic, as if either you understand semantics or you don't. (I'm not saying you're simple, Bob! I think you're brilliant!). My dog seems to understand certain words, but I suspect they are more associations he's learned, rather than a deep understanding of the words. I think a good model for understanding the levels of semantics is studying how semantics develop in children (which is not fully understood). See link #1. I don't know where ChatGPT falls along this continuum, but I'm pretty sure it's not at the top. While listening to the podcast, I kept thinking that what ChatGPT most lacks is metacognition, and without that, you will always end up with some stupid mistakes, as Gary alluded to in his comment about System 1. (In Kahneman's model, only System 2 involves metacognition). Apparently some folks think we can train AI to engage in System 2 thinking (see link #2), even though it currently doesn't think at all, at least if ChatGPT was being honest with me! I'm skeptical AI will ever possess the degree of metacognition that humans are capable of, although not all humans appear to possess it. In working with impulsive children, we teach parents how to be their child's "frontal lobes," and gradually withdraw that support as the child's own frontal lobes develop and he learns to use them. But some never reach a very high level of that!
I have said similar criticisms too JJ in the past, but I think he overplays the statistical parrot metaphor and underplays the ability of Machine learning to do feature extraction and abstraction. A lot of this has to do with confusion about what semantic maps mean and the importance of embodiment and a coherent unified sense of self and purpose in a social world -- and the coming noosphere. I would suggest the following references:
Terrence Deacon on the Noosphere | Closer to Truth Chats
I just listened to Gary Marcus conflate LLMs with text-to-image models, saying that LLMs struggle with negation. Au contraire, LLMs are superhuman negators, while image generators don’t do negation, except with a “negative prompt”.
This relates back to the emergent structure of language, which is sequential and thus amenable and dependent on negation as a means of modifying the meaning of previous statements, and imagery, which is parallel.
One may conceptualize a negative number, but you can’t depict a negative number of objects.
ChatGPT:
Your observation touches on a fundamental difference between language models (like LLMs) and image generation models. Language inherently has a sequential structure that allows for complex modifications, such as negation, to be easily integrated into the processing of information. This makes language models adept at understanding and applying concepts of negation, as the structure of language itself supports this kind of logical manipulation. The sequential nature of language processing means that each word or phrase can alter the context or meaning of what was previously stated, allowing for nuanced interpretations and adjustments like negation.
On the other hand, image generation models work with visual information, which is inherently parallel rather than sequential. This difference in information processing makes it challenging for image generation models to understand and apply negation in the same way. Since imagery is about the presence of visual elements, it's hard to represent the absence (negation) of something without resorting to specific cues or contexts that imply negation. For example, you can't directly depict "not a cat" without showing something else that implicitly suggests the absence of a cat, or by using text in the image to state that negation.
The difficulty with negation in image generation stems from the fact that images represent what is present in a scene, not what is absent. The concept of a "negative number of objects" highlights this issue perfectly; while we can understand the idea of having less than zero of something in abstract or mathematical terms, visually representing this concept involves symbolic or contextual cues rather than direct depiction.
Your distinction between the emergent structure of language and the parallel nature of imagery highlights a critical difference in how these models process and generate content. It underscores the importance of developing models that can handle these unique challenges, especially as we push the boundaries of what artificial intelligence can understand and create.
Marcus, like other scientists steeped in theories of linguistics that emphasize grammar, formal theories of semantics, and symbol manipulation, eg Chomsky, have a serious case of sour grapes.
LLMs confirm the distributional hypothesis, and refute more structural notions of meaning. In a way, it should be obvious that formal grammar is unnecessary — human native speakers learn grammaticality without first learning grammatical rules.
Indeed, my first sentence is misconjugated, and it just doesn't really matter. Indeed, 'misconjugated' is apparently not a legal word, but it just doesn't really matter. Communication transcends grammar.
Bob, your point about semantic space is right on. LLMs help us see that language, rather than meaning something about "the world", works as a vehicle that moves the mind through meaning space. Inherently therefore, the meaning of an utterance depends on the starting point of the listener. Once again, this is obviously true; you have to believe in unempirical academic theories to believe that eg this paragraph has the same meaning to an informed reader and a Chinese child.
Furthermore, predicting the continuation of an utterance is essential to communication! So many people want to poo-poo next-word prediction, but any serious study of rhetoric, humor, persuasion, socialization, etc., has to reckon with the fact that listening to other people talk involves predicting what they'll say next. Before a sentence is finished, before a book is finished, the reader imagines its completion. Entertainment comes from the interplay between confirming and confounding the audience's expectations.
Indeed, the reason that ChatGPT and other LLMs usually produce poor prose is that they are too predictable, they predict the next word too well!
Yes, when he said this I wondered about the different perspectives you might get if you approach it from the LLM as opposed to the image generator POV. If you come from the text-to-image side of things it's harder to imagine any understanding going on, because it screws up so readily, and not just with things like negation, but with even simply concepts. Midjourney gives you six-fingered hands and elbows that bend backwards, from which it's easier to conclude that it understands nothing about anatomy - it's just gluing photos together.
In my listening experience, when a podcast guest describes a tense interview or conversation as "a lot of fun" (or some derivation thereof) it means they are never coming back. I hope that isn't the case here, because I really enjoyed this episode. Civil, mutually respectful, disagreement.... it's a lost art, like silk weaving or coppersmithing.
Hi Bob,
Article suggestion: a synthesis or comparison of Gary and Nora Belrose's take on whether LLM's can "understand" or have "real" semantics. You also have good material from some of these comments here, particularly the ones that are pushing back on your claim that LLM's have understanding or, in the case of your schadenfreude example, have some kind of cognitive empathy. It'd be a good way to clarify the different positions, which I image you'd have to do anyway for the AI book you're writing. Wright or wrong, both the Gary and Nora interviews are gold.
Could not agree more! A fantastic conversation !
It’s never a good time to be Cassandra.
Pointing out ineptitudes of current AI is interesting, so far as it goes, but does little to address doomer concerns. It could be seen as a perpetual act of goalpost-shifting, but definitionally, if it isn’t the AI that causes our destruction, it wasn’t the one we were talking about.
No one thinks GPT-4 is that AI. But many were right to think ‘holy shit’.
It’s not too comforting to know human extinction is difficult; starting over with stones as pillows is better than nothing but it doesn’t fill me with as much optimism as does the starship Enterprise.
What’s required is a little imagination. A little David Deutsch. What is possible with intelligence? What is the limit of intelligence?
How alien it will seem. How unknowable.
It’s never too early to start grappling with the implications. After all, which seems closer, a powerful machine intelligence or effective global governance that might mediate something of similar impact?
We have an imagination AND coordination problem.
The problem with the metaphor is that Cassandra was literally clairvoyant. We’re all just bozos speculating about what might happen.
It’s the bozo part that guarantees the outcome.
Only a true bozo — a veritable schmoe, if you will — would assume that any outcome is guaranteed.
This was an extremely interesting conversation which goes to the heart of a good deal of the internet hype around AI. I listened to it twice. Gary has a deep and nuanced understanding of AI, what it is, what it can do and what it can’t do. Below Bob asks both JJ and me what we think he got “literally” wrong in his talk with Gary Marcus.
The first point that Bob has literally wrong, is his argument that LLMs have an “understanding” of what they spew out. LLMs have nothing that could reasonably called “understanding” of anything that they say or do. As Gary points out repeatedly, an LLM does not know anything about how the world is or operates. It has no judgment, motivation, goals or understanding of the world. What it does, is spew out information based on statistical correlations among words and word usage based on the materials it is trained on. That’s why it delivers so much false, “hallucinatory” and other absurdly wrong information. An LLM has no way to anchor itself in reality, it has no subjective perspective, goals or motivation and is indifferent to what it says and to what you ask it to do.
So Bob is incorrect both on his suggestion that LLMs might have “understanding” (or any kind of sentience, consciousness or subjective experience). LLMs can do what Siri does, such as look up movie times at your local theater but that doesn’t mean it understands what a movie or what a theater is. It is looking information up on the internet as people used to look up words in dictionaries.
This also means that LLM are far less of a threat than Bob thinks. The biggest danger of LLMs is the the fact that people may rely on them, as Air Canada recently discovered to its horror when it was held liable for a non existent discount program that its generative AI made up and convinced customers to use. Gary’s point about self driving cars is also key here. While self driving cars can do amazing things, they don’t work and aren’t reliable and cannot be made reliable because their understanding of the world is too limited to avoid monumentally stupid decisions like driving into an airplane because they don’t know what an airplane is and don’t expect to see it on the road.
The conversation about syntax and semantics was itself really just a semantic argument. It didn’t inform or resolve any issues. What Gary really meant though is that LLMs don’t understand what words mean, they just look at statistical correlations of the occurrence of words. Characterizing what LLMs do as semantic or syntactical is just a not very useful characterization. The real point is that LLMs aren’t sentient and have no what they are doing in relation to the external world. John Searle’s Chinese room experiment isn’t even worth talking about here because it is a lousy thought experiment and it isn’t at all how LLMs work.
Bob is also wrong on the idea of the possibility of implementing an AI slowdown or pause. The whole world is working on accelerating the development of AI and nothing can stop that. The stock market is driving investment of billions in AI research which is the hottest field in business today.
Having the US put a tax on AI would simply cause AI research to move outside the US. And there is no possibility of getting the whole world to adopt such a tax. And if the whole world did in fact adopt a global tax then the big companies that can afford it would simply pay the tax and keep going as they do with every regulatory addition. So the effect would just be to push even more of the development into the hands of big companies that can afford any tax imposed.
So Bob, those are some of the things you were “literally wrong” about in this conversation and why.
So are you defining 'understanding' as entailing subjective experience? If so, I don't think that's a useful definition, since we have no way of knowing whether AIs have subjective experience. And if not, what is your definition of 'understanding'? In other words, what properties would an AI have to have in order to have understanding by your definition? (Also, do I take you to be saying that I've said LLMs have subjective experience? I've never said that and am agnostic on that question.)
Bob,
Thanks very much for taking the time to reply.
Yes the word "understanding" does entail awareness and that involves subjective experience of some kind. If "understanding" doesn't involve "awareness" those terms lose all meaning. And one thing that is very clear is that computing statistical correlations does not in any sense involve "understanding".
So we do absolutely know that LLMs do not have that kind of subjective
awareness or sentience or consciousness or understanding (or whatever you want to call those qualities of mind) since all LLMs do is the computation of statistical correlations.
LLMs can determine that the word "love" is oven used near the words "I" or "you," but they have absolutely no idea what the word "love" actually means.
I would strongly suggest you read Philip Ball's "The Book of Minds" which is available on audible and thoroughly discusses all of these issues and the history of AI, including the move from symbolic to
neural nets. https://www.amazon.com/Book-Minds-Understand-Ourselves-Animals-ebook/dp/B09QD7N55R/ref=sr_1_5?crid=2XVG3YA4PDIO6&dib=eyJ2IjoiMSJ9.RF5mTxB8TZPI1oV4naaK2j_bo9iMqB0ENt4Xv2HHeszUbQJmSbZjVsX00GYp1TgCekOwFIBnAumA2v2zjd8z7ObVeU1_7326gYxvwVLB9bo0Zwzkt5AdQ2WXZVrtChPn7spKdOm1gzoWwMbYnHcDq5IOIFdspwuGl-qoiArhZLVyBeDD6q21MX2yuqgIdoeu8hanyd_dHG_dM2Jmw2rQigMG1OLshkxumfzjhUKqsX0.eTLE16ns1NnYiGrtWRjBn3sadLUyhNGXT5xylR7ptSk&dib_tag=se&keywords=Philip+Ball&qid=1708290633&s=digital-text&sprefix=philip+ball,digital-text,116&sr=1-5
The great chess and Go pjaying computers like Deep Mind and Alpha Go don't even know they are playing a game and don't care if they win. They are just doing computations. The key point is that you cannot understand meaning if you don't have any goals or motivations. LLMs and all existing AIs lack goals or desires and so they don't understand meaning.
If you read Ball's book you will give up your "semantic" argument because it jut makes no sense in light of what LLMs actually do.
And that applies to your schadenfreude example. As Gary points out in your talk, there is no reason to think that isn't just another statistical correlation. If you are embarrassing someone publicly, normally some people in the audience like that embarrassment. It's not at all a rare occurrence.
Thanks for the clarification. I don't agree that we "absolutely know" that LLMs don't have subjective experience. I think there's no way (for the time being at least) for us to know whether they do or don't have it. And that's why, if you're going to define understanding has inherently involving subjective experience, I don't see any point in debating whether AIs have understanding. I elaborate on this in a piece about Searle's Chinese Room thought experiment that I'll post this week. (BTW, I'm not saying it's *wrong* to define understanding as entailing sentience. I'm just saying that if you want to have a useful debate about whether AIs have understanding, you need to choose another definition.)
Bob,
Thanks much for your reply. I am actually making a broader comment.
Subjectivity is fundamental to understanding. I don’t think you can redefine “understanding” so that it can exist separate from awareness and subjective experience. Otherwise the word ‘understanding’ isn’t doing anything in your argument. You cannot have meaning without subjective experience because there is no context for it and meanings are always contextual, not objective. For an information theory approach to the same issue see Christoph Adami’s excellent conversation with Sean Carroll released just this morning.
Link: https://www.preposterousuniverse.com/podcast/2024/02/19/266-christoph-adami-on-how-information-makes-sense-of-biology/
Adami ’s approach as well as Philip Ball’s explicit statements agree with mine and are worth your having a look when you have a moment.
On the point that "[o]therwise the word 'understanding isn't doing anything in your argument," let me pose a related question to Bob: have you identified any example of AI behavior in which statistical pattern recognition/application alone would not explain the outcome--thus revealing that "something else" is happening in these systems? As a number of us have noted, your schadenfreude example is not actually an example of that. Do you have any others--i.e., examples of AI behavior that reveal something more than mechanical pattern recognition/application? If so: (a) what are they, (b) why do you believe they show something more than pattern recognition/application, and (c) and what is your explanation for those examples, at a nuts-and-bolts level? Unless you can provide such an example to illustrate your position, it's quite unclear what you are referring to vis-a-vis "understanding."
Hi Bob,
This was a great conversation, thanks very much.
I have the greatest admiration for you as a public thinker. On topic after topic over the years, I have been so impressed by your perspectives and positions, and have almost always agreed with you.
Until generative AI. As I wrote in some comments I posted in the "Parrot Room" last year, many of your positions on generative AI simply don't reflect how the technology actually works. Marcus did a terrific job explaining some of the key errors in your positions. I had tried (obviously unsuccessfully) to make some of the same points in my posts. For example, I discussed the same "shadenfreude" episode, and like Marcus tried to explain why it didn't mean what you think it means.
I really wish you would step back and reappraise your positions on generative AI. I have the sense that you jumped to some conclusions early on, and ever since have been looking for reasons to defend those early positions. It's of course hard to admit error, but it's worse to stubbornly stick with positions that are not correct.
To be clear: much of what you say about AI is indeed correct--its massive power, its significant potential downsides, the need for great care in its development and deployment. But you are continuing to stick with a conception of how it works--at the mechanical level--that is simply not correct, and that is detracting from your larger commentaries on AI.
I would love to hear how you would grapple with the many fascinating implications of how generative AI actually works. That would be worthy of the book that you are writing--there are so many fascinating issues, ranging from policy questions to philosophical questions, that generative AI presents. But to truly confront those questions would require recognizing how generative AI actually works.
Thanks for your comment. What did I say in my conversation with Gary that was literally "incorrect"?
Bob,
After generative AI became a big topic last year, you quickly developed a position against the "supercharged auto-complete/stochastic parrot" explanation of generative AI. You criticized that explanation as missing something deeper going on with generative AI.
I think you developed that position because you viewed the auto-complete/stochastic parrot explanation as somehow denigrating the power of AI and ignoring its potential consequences. But whatever your motivation, you took that position early, and have since dug in deeper and deeper.
The position is simply not correct. Let me make a few points (versions of which I made in the Parrot Room comments last year):
First, generative AI is fundamentally about statistical pattern recognition (in data sets) and pattern application/reconstruction (in response to user prompts)--just like autocomplete. The power of generative AI systems dwarfs what we had known as simple autocomplete, but the basic approach of identifying statistical relationships among characters in data sets, and then applying those patterns to user prompts, is the same. Terminology like "vectors" and "neural networks" can sometimes confuse this, because the words may suggest something else is going on. But if you strip away the jargon, and examine the actual mechanics, both generative AI and autocomplete are forms of pattern recognition/pattern application. Gary was correct to make the autocomplete comparison in your discussion with him.
Second, this is profoundly different than systems that reason from first principles, as Gary pointed out using a variety of examples. The generative AI systems are simply parroting language based on statistical relationships, not engaging with the underlying ideas represented by that language. That is why the "stochastic parrot" characterization is also correct.
Here is another example: imagine a society that has a well-developed understanding of math, and expresses their correct understanding in various writings about math. But, for whatever reason (this is a thought experiment), the society has a glitch--every member in that society believes that 2 + 2 = 5, and every member in that society reflects that error in their writings. This society would create a set of writings that (a) accurately describe abstract mathematical principles but (b) reflect the society-wide glitch in believing that 2 + 2 = 5.
If a generative AI system trained itself on the writings of that society, it would be able to recite math principles correctly--because the writings of the society properly described the abstract principles. But if you asked that AI system "what is 2 + 2?", the response would be "5." That is because generative AI is simply parroting the results of the patterns that it has detected, and in this thought experiment, the uniform, society-wide pattern was to say 2 + 2 = 5. The fact that this AI could also recite the abstract mathematical principles correctly does not mean it is "learning" those principles in the way that we do--as Gary explained--and does not mean it could use those principles to correct the society-wide error in believing 2 + 2 = 5. It couldn't, and it wouldn't.
Now, we use words to express our ideas, and our use of words create linguistic patterns, such that AI "parroting" the words in those patterns may give the appearance that the AI understands the ideas underlying the words. As you have noted, there are amazing examples of AI's generating texts that seems consistent with our human understanding of the meaning of the ideas underlying the words. But again, that is just a function of the statistical relationships among the words in the data set. Put another way, it is no surprise that AI systems will sometimes use words in patterns that seem consistent with our understanding of the ideas underlying the words--after all, the AI systems train on data sets that reflect humans expressing their ideas in words, which create patterns of language consistent with those ideas. But as Gary pointed out through several examples, parroting language that sometimes seems consistent with our understanding of the underlying ideas is very different than reasoning based on the underlying ideas--by, for example, applying those underlying ideas to contexts outside the original data set. (See his example of the Tesla running into the jet.)
I have the sense that your schadenfreude episode (which you have cited repeatedly) played a significant role in leading you astray on how generative AI works. You seem to have interpreted that as a remarkable example of AI understanding the subtle concept of schadenfreude, and applying it to a new setting (the one you described in your prompts). That's not correct. What was actually happening is that the AI system had identified subtle linguistic patterns in its training data involving the word "schadenfreude," and identified elements of those patterns in your prompt, such that it listed schadenfreude as a possibility. The AI doesn't actually know what schadenfreude means. You mistook its (incredible) power of linguistic pattern recognition/application to be an understanding of the underlying meaning of the word. You were wrong about that, and you've made similar errors in much of your commentary about AI.
Third, you sometimes (including in your conversation with Gary) say "well, we don't really know what is going on inside these systems," which I take to be a suggestion that there is something deeper occurring than pattern recognition/reconstruction. But we *do* know what is going on in these systems, in terms of the basic mechanics. When people talk about the "unknowns," what they are referring to is that the AI systems are picking up on subtle and nuanced linguistic patterns that (at least initially) are not discernible to humans. That is truly remarkable, and part of why generative AI is an amazing technology--the machine learning on enormous data sets yields pattern recognition that would have been unimaginable in the past, and that (as Gary said) is hard for humans to comprehend now. But that does not mean the basic mechanics are unknown--they are completely known. Put another way, we know how the machines are designed and how they operate--but the statistical patterns generated by those operations can be hard to comprehend. Yet even though they can be hard to comprehend, they are just statistical patterns among text characters.
Fourth, as I said above, saying that generative AI is "just" statistical pattern recongition and pattern application/reconstruction is, in no way, to denigrate the power of the technology. It is immensely powerful, and I share many of your concerns--e.g., the risk of bad actors using AI to proliferate disinformation. But it is powerful in a somewhat different way than you have suggested, and recognizing what it actually is has implications for the policy and even philosophical issues posed by AI.
Fifth, I encourage you to continue having conversations with folks who have similar views as Gary. For example, I'd love to hear you have a discussion with Emily Bender, one of the authors of the article "On the Dangers of Stochastic Parrots."
Finally: there are so many truly fascinating questions presented by AI, ranging from policy issues to deep philosophical questions. I tried to list some of those in one of my Parrot Room posts. I can think of no one that I'd rather see grapple with those questions than you. But you're just not doing it at the moment, because you set off early on a path that misconceives what generative AI is, and without a clear understanding of what it is, you can't properly engage with the true policy/philosophical implications of AI.
You make no mention of the fact that LLMs create a massive multidimensional 'map' of words in a kind of 'semantic space.' To me, that's the first thing to take into account if you're going to have a serious discussion about whether, and in what sense, LLMs 'understand' things. And to you it's not even worth mentioning? In any event, I emphasized that fact in my conversation with Gary, so if you're going to do a serious critique of my position, I don't see how you can ignore it. You keep repeating that the LLMs just sense 'statistical patterns' without taking into account the internal structures that the models build in order to be able to do a good of handling the statistical patterns. And again: We *don't* know everything about the internal structures yet. But we do know about the semantic mapping. If you don't think the semantic mapping is even worth mentioning, you and I see this whole thing so differently that I doubt further discussion of this would be productive. But I do appreciate your taking the trouble give me the feedback.
Bob, the "map" is comprised of the patterns of statistical relationships among the elements in the data set. It is indeed remarkable--that is precisely what I have said (repeatedly) in noting the immense power of LLM's to discern subtle and nuanced patterns in the data set. But the map is not more than those statistical patterns. Using the term "semantic space" is just deploying jargon, and adds nothing to the substantive discussion of the mechanics of these systems.
I've tried several times, including in my Parrot Room comments, to explain what I see as the problems in your perspective on generative AI. I either am not articulating these problems well enough, or you simply don't agree that they are problems. Either way, I'm going to throw in the towel on this discussion.
My last comment is I meant what I said: I can think of no one that I'd rather see grapple with the policy and philosophical implications of how AI actually works, than you. But to do that, you're going to need to admit some mistakes and move on. Nothing wrong with that! These are complicated issues, and having some early misunderstandings is no big deal. But sticking with those misunderstandings is going to continue to detract from your commentaries on AI.
I fully understand the sense in which what I call a mapping of words onto semantic space is based on statistical patterns in language. What I don't understand is why you object to the term 'mapping of words onto semantic space.' After all, those statistical patterns are to some extent reflective of the human understanding of the meaning of the words in the first place; so why should it not be possible to extract something from those patterns that warrants the term 'semantic'? Dismissing the term 'semantic' as 'jargon' is just name calling. You need to make an actual argument as to why the term doesn't apply.
My objection is that you seem to be using the term to suggest the "map" entails--at the mechanical level of how these systems work--more than the detection of statistical patterns in language. It doesn't. In terms of the mechanics of these systems, they are just using statistical patterns in language. That's it.
You are certainly correct that these patterns have some correlation to what humans perceive as the underlying meaning of those words, such that the AI's use of those patterns will give the appearance that the systems understand the meaning of the words (at least sometimes). In one of my earlier posts, I had a paragraph on exactly this. (It starts with this sentence: "Now, we use words to express our ideas, and our use of words create linguistic patterns, such that AI "parroting" the words in those patterns may give the appearance that the AI understands the ideas underlying the words.")
But this correlation does not mean the systems actually understand the underlying meaning of the words--they don't. You asked Ted Farris whether this question of "understanding" turns on whether the AI systems have subjective experience. I think subjective experience is part of it--I think subjective experience is an important part of how we engage with the underlying meaning of words. But as you note, we cannot prove the AI systems lack subjective experience. But I think proving the AI systems do not have "understanding" of underlying meaning does not require reliance on our intuitions about whether they have subjective experience. It can be proven empirically, including through real-world examples like the ones that Gary provided in your discussion, in which the AI systems do things that they would not do if they had command of the underlying meaning of the words they are using. And, I think my "2 + 2 = 5" thought experiment in one of my earlier posts makes the same point.
I also agree with Ted Farris that ultimately, whether we call the "map" a "map of semantic space" is itself a semantic issue. That is what I meant when I said deploying jargon doesn't add anything to the substantive discussion of the mechanics of these systems. One of the things that I've always loved about your writing and your discussions is that you have a gift for explaining complicated technology (e.g., quantum mechanics) in simple, plain English terms. The reason I've appreciated that approach is because it strips things down to the core issues, and allows those issues to be discussed without the distractions of constantly defining terms, etc.
Here, we know--and you acknowledge--the AI systems are detecting statistical patterns among characters in data sets. That's it, as far as I'm concerned--there is the detection of those patterns, and then the application/reconstruction of the patterns in response to user prompts. (As I have said over and over, "that's it" doesn't mean these are anything other than wondrously powerful---they are indeed wondrously powerful.)
If you think something else is going on beyond that, what is it? And how would you explain that without resorting to jargon, and without saying "we just don't know what is going on in these machines"? We *do know* the systems are developing and applying statistical models. If you think--or even just have a hunch--that there is something more than machines doing pattern recognition and pattern application, please explain what it is, in clear terms.
My principal problem with your approach to this subject is that, starting last year, you seem to have had an initial reaction that these systems must be doing more than pattern detection / pattern application--and ever since then, you seem to be looking for ways to defend that initial reaction. It's a fruitless quest: they are not doing more than pattern detection / pattern reconstruction. And I think if you try to take your best shot at answering what "what else are they doing beyond that?" in plain English, this would actually help show the problems with your position, by exposing the core issues stripped of jargon.
As I've said over and over, there are so many fascinating questions that remain, even if you were to accept that these systems are "just" doing linguistic pattern recognition / pattern application, without command of the underlying meaning of words. But you can't truly engage with those questions without accepting the reality of how these systems work.
Very well said JJ. I will comment further in due course. Gary's objections to many of Bob's positions were very well and precisely phrased and show a nuanced judgment that Bob can't seem to hear. Bob is sticking to positions he has had since ChatGPT4 was originally introduced which are both wrong and result from cognitive biases toward seeing an apocalypse, disaster or existential threat in most every significant technological or political development and seeing global governance (always impractical) or a stop to irresistible market or political forces as the only solution.
Which of my positions are wrong--and why are they wrong?
I've got an answer for you from your most recent 'Earthling Unplugged': "The whole thing about AI is it can in principle, as it evolves, do everything people do with their heads."
LLMs can do some things we do 'with our heads,' like manipulate language, get right answers to questions, notice and produce patterns, etc. But there is a huge class of things we do with our heads, to me the most important things, that it is *in principle* not capable of doing: it can neither understand nor express meaning.
[I know you are very impressed by what might be called a 'semantic map' found in LLMs -- that dog is mapped closer to cat than to aircraft carrier, etc. But nothing about the ability to reproduce accurate semantic patterns of this sort from huge swaths of text suggests actual semantic engagement (actual understanding/expression of meaning).]
None of this is to say we should stop worrying about the potential enormous harms of AI - bad actor stuff, job replacement stuff, etc. But I do think that we should START worrying about the discourse around AI collapsing or deflating how we view our own human capacities for understanding and expressing meanings.
Case in point, from your comments:
"LLMs help us see that language, rather than meaning something about "the world", works as a vehicle that moves the mind through meaning space. Inherently therefore, the meaning of an utterance depends on the starting point of the listener. Once again, this is obviously true; you have to believe in unempirical academic theories to believe that eg this paragraph has the same meaning to an informed reader and a Chinese child."
This is a view of meaning that is strikingly devoid of human agency. If what something means is simply 'what-it-is-taken-to-mean,' misunderstanding (and thus, understanding) is impossible.
I listen to a large number of podcasts across a wide spectrum of topics. I dislike the amateur podcaster mode of interviewing of talking way too much in the pretence of conversation. In this case Robin Wright sounded like a petulant child and I stopped halfway through despite being very interested in the subject and interviewee. I long daily for better interviewers to emerge but in this era of unbridled expressionism I fear this is another skill lost in the digital sands
My name's not Robin.
Great discussion into an amazing and frightening world. All artificial intelligence should be subject to strict regulations by none corrupt politicians and Ai researchers.
Well that was good, and surprisingly civil.
My take is that this "pattern-matching" vs "understanding" thing is a false dichotomy. They are the points on a spectrum, and when we say "pattern-matching" we really mean the same thing as "shallow understanding".
Marcus claims that AIs fall apart when you step "outside the bounds" of the training data. But strictly speaking - almost everything is "outside the bounds" of the training data. Most sentences, including the one you are reading right now, have never been written down before. LLMs have clearly demonstrated an ability to operate outside "the bounds" - in that they respond in a reasonable way to many questions they have never seen before. The real question is "Can they reach human-level out-of-boundsness using current approaches - or is something fundamentally different needed"? THAT is where the debate could potentially be productive.
The examples Marcus gave are kind of questionable. The ones I've seen so far are:
GPT knows who Tom Cruise's mother is (Mary Lee Pfeiffer), but not (in an independent conversation) who Mary Lee Pfeiffer's son is (have confirmed - it's legit).
- This seems to just say that the model can contain knowledge that it cannot recall - humans do do this all the time - forget facts and are able to recall them later when queued correctly. I don't see how it indicates a lack of "understanding" on the AI's part - just that knowledge storage is asymmetric.
Hallucinations about Gary's pet chicken / Elon's car crash, etc.
- These are a little more interesting - but in the examples, the model was just prompted to say something - not told that it needed to be true. There is something interesting their about the model's inability to understand when it doesn't know something - but humans have similar tendencies (the term is "bullshitting"), albeit usually less extreme. Still - this doesn't seem to indicate a categorical difference in the way these things process information.
Math: inability to infer and execute algorithms.
- It's probably true - you can train these things on adding 10-digit numbers all day and they'll probably never get it right, despite begin able to recall the algorithm if prompted! (actually no longer true - the new GPT can write and execute code, and will correctly say that "The sum of 9576105326 and 3827630678 is 13,403,736,004"). But I'd argue the way the model responds (pre-code executor) is more or less how a human would respond if you asked them to calculate the result without pen or paper and they had to answer on the spot. GPT4 (like most humans) does not have an internal notepad to write down intermediate results and perform recursive algorithms on. But recursive thinking is not out of the realm of AI (just look at AlphaGo) so maybe the marriage of these approaches is the ticket.
In short - all these "counter examples" are things that humans do to some degree - none of them seem to me to indicate a fundamentally different kind of thing going on under the hood.
edit: I forget the "Draw me a room with no elephants in it". This one is actually more interesting - you still do get rooms with elephants. It seems to be more about a deficiency in DALL-E (the image-generation model) than GPT. It seems that the image-generation model doesn't really understand language - just associations of words and imagery coming from the captioned images it's trained on. GPT itself seems to have no problem with "Write me a limerick with no elephants in it".
Good discussion. This is the most subtle I've heard Gary Marcus be. Really liked his comparison to GPS systems. Re the intentionality/aboutness issue, I'd love for Bob to talk to an eliminativist like Alex Rosenberg to hear what he's thinking about AI and what it means for intentionality. Also, Raphaël Millière would be another great guest to talk about AI and philosophy. He was superb on Sean Carroll's Mindscape podcast and Bob could have a fruitful discussion with him.
Unlike most of your other commenters, I'm not an expert in how AI works; I'm just a lowly psychologist who "understands" a little bit about the still mysterious human mind. Given that AI is basically a highly sophisticated and sensitive correlation detector and nothing more, it seems unlikely to me that they "understand," at least in the same manner humans do, including ascribing significance. To think that they do reminds me a bit of the mistake some people make in thinking correlation = causation. But just to be sure, I went straight to the horse's mouth and asked ChatGPT, and here's some of our conversation, edited for brevity: Me: Do you possess understanding? ChatGPT: I don't possess understanding in the way humans do. I don't comprehend the information or have consciousness. My responses are generated based on patterns learned from data, and I don't truly understand the meaning of the information. I don't have awareness, beliefs, or subjective experiences. If my responses seem like understanding, it's a result of patterns and associations present in the data on which I was trained. Me: Can you think critically? ChatGPT: No, I don't have the ability to think critically. While I can provide information, analyze patterns in data, and generate text based on learned information, I lack true understanding, consciousness, and the ability to engage in critical thinking. Critical thinking involves evaluating information, considering different perspectives, and making reasoned judgments, which goes beyond the capabilities of current AI models like mine. Me: Do you possess metacognition? (Note: I don't think you can possess meaningful understanding without metacognition, and it's why ChatGPT sometimes makes very stupid mistakes). ChatGPT: No, I don't possess metacognition. Metacognition involves the ability to reflect on one's own thoughts and cognitive processes. While I can generate responses based on patterns learned from a diverse range of data, I do not have self-awareness, introspection, or the ability to think about my own thinking. Me: Why do people anthropomorphize you? ChatGPT gave a lengthy answer that described four major reasons why humans might anthropomorphize ChatGPT, including this: Some users might not be fully aware of the limitations of artificial intelligence models like mine. They may project human characteristics onto me due to a limited understanding of the underlying technology.
So there you go! When it comes to semantics, it seemed like you possess a somewhat simplistic representation of a very complex topic, as if either you understand semantics or you don't. (I'm not saying you're simple, Bob! I think you're brilliant!). My dog seems to understand certain words, but I suspect they are more associations he's learned, rather than a deep understanding of the words. I think a good model for understanding the levels of semantics is studying how semantics develop in children (which is not fully understood). See link #1. I don't know where ChatGPT falls along this continuum, but I'm pretty sure it's not at the top. While listening to the podcast, I kept thinking that what ChatGPT most lacks is metacognition, and without that, you will always end up with some stupid mistakes, as Gary alluded to in his comment about System 1. (In Kahneman's model, only System 2 involves metacognition). Apparently some folks think we can train AI to engage in System 2 thinking (see link #2), even though it currently doesn't think at all, at least if ChatGPT was being honest with me! I'm skeptical AI will ever possess the degree of metacognition that humans are capable of, although not all humans appear to possess it. In working with impulsive children, we teach parents how to be their child's "frontal lobes," and gradually withdraw that support as the child's own frontal lobes develop and he learns to use them. But some never reach a very high level of that!
https://sites.socsci.uci.edu/~lpearl/courses/readings/Wagner2010_AcqOfSem.pdf
https://philarchive.org/archive/BERTFA-9
I have said similar criticisms too JJ in the past, but I think he overplays the statistical parrot metaphor and underplays the ability of Machine learning to do feature extraction and abstraction. A lot of this has to do with confusion about what semantic maps mean and the importance of embodiment and a coherent unified sense of self and purpose in a social world -- and the coming noosphere. I would suggest the following references:
Terrence Deacon on the Noosphere | Closer to Truth Chats
https://www.youtube.com/watch?v=aBNmAleGbnE
Signs of Artificial Life Symposium | Day 1 with Terrence Deacon
https://www.youtube.com/watch?v=JhXALo0Ri7Y
Jaron Lanier Looks into AI's Future | AI IRL
https://www.youtube.com/watch?v=o8vU9QtcMZw&t=26s
Intelligent Thinking About Artificial Intelligence
https://www.youtube.com/watch?v=caepEUi2IZ4&t=319s
Me:
I just listened to Gary Marcus conflate LLMs with text-to-image models, saying that LLMs struggle with negation. Au contraire, LLMs are superhuman negators, while image generators don’t do negation, except with a “negative prompt”.
This relates back to the emergent structure of language, which is sequential and thus amenable and dependent on negation as a means of modifying the meaning of previous statements, and imagery, which is parallel.
One may conceptualize a negative number, but you can’t depict a negative number of objects.
ChatGPT:
Your observation touches on a fundamental difference between language models (like LLMs) and image generation models. Language inherently has a sequential structure that allows for complex modifications, such as negation, to be easily integrated into the processing of information. This makes language models adept at understanding and applying concepts of negation, as the structure of language itself supports this kind of logical manipulation. The sequential nature of language processing means that each word or phrase can alter the context or meaning of what was previously stated, allowing for nuanced interpretations and adjustments like negation.
On the other hand, image generation models work with visual information, which is inherently parallel rather than sequential. This difference in information processing makes it challenging for image generation models to understand and apply negation in the same way. Since imagery is about the presence of visual elements, it's hard to represent the absence (negation) of something without resorting to specific cues or contexts that imply negation. For example, you can't directly depict "not a cat" without showing something else that implicitly suggests the absence of a cat, or by using text in the image to state that negation.
The difficulty with negation in image generation stems from the fact that images represent what is present in a scene, not what is absent. The concept of a "negative number of objects" highlights this issue perfectly; while we can understand the idea of having less than zero of something in abstract or mathematical terms, visually representing this concept involves symbolic or contextual cues rather than direct depiction.
Your distinction between the emergent structure of language and the parallel nature of imagery highlights a critical difference in how these models process and generate content. It underscores the importance of developing models that can handle these unique challenges, especially as we push the boundaries of what artificial intelligence can understand and create.
Marcus, like other scientists steeped in theories of linguistics that emphasize grammar, formal theories of semantics, and symbol manipulation, eg Chomsky, have a serious case of sour grapes.
LLMs confirm the distributional hypothesis, and refute more structural notions of meaning. In a way, it should be obvious that formal grammar is unnecessary — human native speakers learn grammaticality without first learning grammatical rules.
Indeed, my first sentence is misconjugated, and it just doesn't really matter. Indeed, 'misconjugated' is apparently not a legal word, but it just doesn't really matter. Communication transcends grammar.
Bob, your point about semantic space is right on. LLMs help us see that language, rather than meaning something about "the world", works as a vehicle that moves the mind through meaning space. Inherently therefore, the meaning of an utterance depends on the starting point of the listener. Once again, this is obviously true; you have to believe in unempirical academic theories to believe that eg this paragraph has the same meaning to an informed reader and a Chinese child.
Furthermore, predicting the continuation of an utterance is essential to communication! So many people want to poo-poo next-word prediction, but any serious study of rhetoric, humor, persuasion, socialization, etc., has to reckon with the fact that listening to other people talk involves predicting what they'll say next. Before a sentence is finished, before a book is finished, the reader imagines its completion. Entertainment comes from the interplay between confirming and confounding the audience's expectations.
Indeed, the reason that ChatGPT and other LLMs usually produce poor prose is that they are too predictable, they predict the next word too well!
Ah, the incongruity theory of humor! Now I know why ChatGPT just isn't that funny! :)
Yes, when he said this I wondered about the different perspectives you might get if you approach it from the LLM as opposed to the image generator POV. If you come from the text-to-image side of things it's harder to imagine any understanding going on, because it screws up so readily, and not just with things like negation, but with even simply concepts. Midjourney gives you six-fingered hands and elbows that bend backwards, from which it's easier to conclude that it understands nothing about anatomy - it's just gluing photos together.