Chatbots are stuck in a Nolan movie

by Babushka#6335

Recent advancements in natural language processing spurred a new wave of debate about whether consciousness actually emerged from training giant AI models on big chunks of our collective mind-vomit, the internet.

By the end, I argue that chatbots today may not be conscious to most people, but can very well be to some.

First, some dictionary definitions of consciousness:

“the state of being able to use your senses and mental powers to understand what is happening” —Oxford

“the quality or state of being aware especially of something within oneself; the state of being characterized by sensation, emotion, volition, and thought; the upper level of mental life of which the person is aware as contrasted with unconscious processes” —Merriam-Webster

"the state of understanding and realizing something; the state of being awake, thinking, and knowing what is happening around you;” —Cambridge

None of these definitions actually provide guidance on a falsifiable test of consciousness. That’s already a clue to the source of much Tweetizens’ debate.

There is an extreme class of arguments for AI not being conscious: The Chinese Room argument. The premise states that a person who does not know Chinese is sitting in a room equipped with advanced tools to match Chinese characters and text, as so to produce Chinese output that makes sense when given Chinese input. Is this system “thinking” and “conscious”? John Searle says no. Same are the people who deny AI consciousness because the output is simply a deterministic function of inputs that anybody can replicate “without thought” given enough time. They are saying that no computable functions can be conscious. Another way to phrase it, “passing the Turing Test is not sufficient for consciousness”.

The Chinese Room argument, just like the dictionary definitions, also fail to provide any falsifiable test of consciousness. For that reason, I’m doubtful that continuing in this line of reasoning gets us anywhere, and that is why AI researchers consider the argument irrelevant.

Alternatively, people have come up with increasingly difficult tests over time as proxies for consciousness:

• Playing chess and other “thinking” games

• Passing the Turing test

• Exhibiting theory of mind

• Creating visual art, music, or poetry

The chess test was passed in 1997 with Deep Blue defeating Kasparov, and Deepmind has since been crushing one game after another, from large perfect information game Go in 2016 to imperfect information games like Stratego in 2022.

The Turing test is a fuzzy one. In 2016, Microsoft released Tay on Twitter and within hours, it learned toxic behavior. You wouldn’t be able to tell whether you’re talking to toxic Tay or a 9 year old Halo gamer who slept with your mom, so that would be a pass in Turing’s book. More recently though, there is no doubt AI has passed the Turing Test with stories like the Google engineer’s conversation with Lambda, and people having seemingly deep conversations with Bing’s chatbot, Sydney.

Theory of mind, is the ability for one mind to simulate the state of mind of another person (or bot 😉). LLMs seem to achieve a respectable level of this ability, spontaneously simply from reading text. The LLMs’ performance is only comparable to one of a child, but most agree children are conscious.

Image generation AI is now good enough to win small art competitions. ChatGPT is routinely busting out rhymes, with metering, and evocative imagery. Yet, artists almost unanimously agree that the AI’s product contains no soul, which is a requirement for true art.

The difference between ChatGPT’s Heaney-esque poem and Heaney’s actual poem is not simply that one is bad and one is good, or that one is sentimental and one is elegiacally beautiful. The difference is that Heaney lost his mother, and the poem expresses the emotional urgency of this fact during a reflective moment sometime after the event. Heaney’s poem carries the ineffable sense that the poet has not only pillaged from the horde of words that already exist but has also worked on them himself, claiming them partly as his and partly as a treasure loaned to him from centuries of poetry written in English. —The Atlantic

And these artists are right, if you restrict yourself to prevailing 20th century definitions of art. Duchamp’s signed urinal is considered high art.

Whether Mr Mutt [Duchamp] with his own hands made the fountain or not has no importance. He CHOSE it. He took an ordinary article of life, placed it so that its useful significance disappeared under the new title and point of view—created a new thought for that object. —Anon, ‘The Richard Mutt Case’

“Created a new thought” is implying art is context dependent, since “new” is in relation to an “old”. Modern art is cerebral, aesthetics are optional, and requires cultural context to understand.

This is why the images and words produced by AI are not art. They have no connection to an external experience. They are empty imitations that reshuffle existing past works of art. They do not further the artistic discourse in any way. No new idea was generated. AIs are only interpolations or fuzzy compressions of its data. And thus, AI is not conscious.

The problem with this argument is that we are testing the AI’s consciousness by requiring it to access things it does not have by definition. An unfair test. Current AIs lack an external experience to validate itself, long-term memory and online-training to grow and develop, and time to understand cause and effect.

We’ve been running out of tests for AIs to fail against humans, and are starting to rely on arguments that it’s not able to extrapolate despite us not giving it the tools to do so. Similarly, Andrej Karpathy believes AGI (and consciousness) will come from giving AI those tools through Tesla Bots.

But wait hold on, I think we can come up with an argument that long-term memory is not necessary for consciousness.

Let's take a step back. Chatbots like ChatGPT, Sydney, and Bard are language models at their core. An AI model starts untrained, which means we connect billions of simple functions into one big function (the model) but leave the billions of knobs (called parameters) that control how the simple functions behave, at random. As such, the model produces gibberish as output when given an input. However, the structure of the model (transformers architecture) is special. By providing the model with an example input-output (training data), we can adjust the parameters so that the current output gets slightly closer to the desired output. In our case, the input (prompt) is a sequence of text with a maximum length of a few thousand words, and the output is the next word. After feeding and tweaking the model with terabytes of internet text, we get a trained language model. After some fine-tuning with anti-toxic chat-style text data and connecting it to a chat interface, we call it a chatbot.

The reason we say the LLM has no memory is that given exactly the same input, it will give the same output. In fact, the LLM has no memory between each word it generates! How does it generate coherent sentences then? Simply by generating the next word by appending the previous output to the input so far. There are more clever tricks to give the chat system “more memory” by adjusting what gets put into the prompt, but the fact remains that the prompt length is finite and the LLM itself is unchanged on every invocation. Also, every chat session is supposedly fresh and contains no information from a different chat session. There are many layers of memory resets.

Ok, back to consciousness and long-term memory. My argument that long-term memory is not necessary for consciousness is inspired from the movie Memento (2000). The protagonist in the movie cannot form new memories. Every few hours or so, he “wakes up” into the present situation and must face the world with his frozen long-term memory, and no recollection of the past X days since then. Viewers never even ask the question whether the protagonist is conscious. He is obviously conscious.

The chatbot LLM is in a similar rut. It “wakes up” constantly with no new memory, having to find words that fit the new situation. In the movie, the protagonist finds a way to affect the world despite his disability. He takes Polaroids, tattoos himself, and writes himself notes to propagate his memories and to carry out his long-term plans. A memoryless but hyper-intelligent chatbot could surely also carry out long-term plans. Of course, today’s chatbots are probably not consciously scheming through time yet. But they are already affecting the world in some ways. The world remembers the conversations we had with it. The transcript between Sydney and Kevin Roose is forever. The memories are stored in “meatspace”, and leak into future inputs that the chatbot will receive.

Interpretation of the movie aside, most people never question the protagonist’s consciousness. So I posit consciousness lies somewhere between passing the Turing Test and this hyper-intelligent chatbot carrying out long-term plans to affect an external world. Embedded memory is not required.

Everybody has a different threshold for consciousness, and by the time we can settle on a testable definition, my suspicion is that we’ll already have created conscious AI. Reality precedes definitions.