In the second scenario, there is no “native speaker” that is telling you what to write. There is a list of instructions that you are following that is the same as the instructions that the computer was following with the first scenario.
The argument is that if, by following the exact instructions that the computer is following, you still do not understand the Chinese conversation, why would you say that the machine understands?
I haven’t thought about the idea enough to say I agree or disagree, but I hope I understood your comment and that the explanation helps.
I’d like to add a rebuttal I tend to agree with: You’re only a component of a larger system, and the whole system (you + the room + your list of instructions) does encode some understanding, otherwise a chinese speaker wouldn’t be able to interact with the room in any meaningful way.
In almost any system, you can look at a specific part and argue that part doesn’t fulfill a property that the entire system taken as one does.
But in this case you is unnecessary if a mechanism can perform that part, and understanding then completely doesn’t rely on consciousness. This is a bit weird way to think about it, but it seems to be a semantic difference
No, you’re still a necessary part of the system, since otherwise there is no interaction possible. And yes, I don’t think understanding relies on consciousness.
Difference in if that is required and if ‘understanding’ is a quality that a system can have. For instance I feel that a gravity coin sorter is made with understanding but does not have understanding
It’s not really possible to give a concrete answer without a concrete definition of “understanding”. But unless you assume supernatural explanations, either systems must be able to “understand”, or humans are also incapable of “understanding”, since we ourselves are physical systems too.
For instance I feel that a gravity coin sorter is made with understanding but does not have understanding
I get the example and agree that such a coin sorter does not meaningfully have an understanding of gravity. But IMO it does have a (minimal) understanding of the different coins you’re sorting, since it contains a mechanism for differentiating between their sizes.
Imagine we replaced the gravity mechanism with a bunch of actuators controlled by organic neurons, which are connected to something like an eye. The neurons fire and cause the actuators to sort the coins. Now, if say the neurons just react to brightness levels it’d be hard to argue there’s an understanding. But if the neurons are part of a real human brain, they can definitely understand the coins etc. and sort based on their understanding.
As you take away more parts of said brain, when does it lose the understanding?
No, it doesn’t make any sense at all. If I couldn’t understand when a native speaker was replying, how does me not understanding when a neural network does it make it any less significant?
In the Chinese Room scenario, cards with Chinese written on them are inserted into a box.
Inside there is a man who doesn’t understand Chinese but has a lookup list of which symbols to write as a response to every possible sentence you can write on the inserted card.
For anyone observing from outside the box it looks as if whatever is in the box is fluent in Chinese and can hold a conversation. But the man inside does not understand anything about Chinese.
This is used as an analogy to explain that machines, very much like the man in the box, do not actually understand the data they are fed, nor the data they output, no matter how human the response might seem.
It was an argument against the Turing test, which is about whether machines showing behaviour indistinguishable from a human is a sign that they can think.
The Chinese Room argument is that they must also show understanding of the data itself, rather than just human-like responses, to say that they think.
An LLM is a series of immutable nodes that pass results on to the next series of nodes until an output is generated, a functionality that could be perfectly imitated with a very large number of people seeing a series of values passed to them from other people, doing a bit of rigidly defined basic math with them, and passing it on to the next set of people who play the part of the next set of nodes.
The fact that these nodes are immutable, have no persistent state, and only process information in one direction (input to output, and no more) means that an LLM has no persistence of experience. You could potentially argue that the information, as it exists, is a form of fleeting consciousness, but it would be so incredibly alien compared to the lived experience of humans that it would be impossible to relate to in any meaningful way - and it would also mean that every time an LLM is run, the consciousness is created and destroyed for every single word that it outputs… unless you can somehow argue that the consciousness is contained in the set of tokens that gets fed back into the LLM with each word it outputs, which would also mean that any string of text is the equivalent of latent consciousness, which is patently absurd.
You could potentially argue that the information, as it exists, is a form of fleeting consciousness, but it would be so incredibly alien compared to the lived experience of humans that it would be impossible to relate to in any meaningful way
Curiously, one of the approaches to the consciousness, and a more Zen-ish one, is that clinging to a persistent ‘self’ is erroneous and it’s better seen as a process rather than an entity. Granted, it still assumes a ‘self’ that is changed by each passing experience.
Consciousness is a discrete phenomenon that exists, and not all things are conscious. A piece of paper, for example, is not conscious.
A string of text can be used to represent any type of data. It would mean that every piece of data is a consciousness, which - if we assume item 1 to be true - cannot be true, as a piece of paper (which is not conscious) can hold the data.
It’s a discrete phenomenon because my consciousness is not your consciousness. I do not have two consciousnesses at once. Each consciousness is separate from any others - hence, discrete.
As for how I know another person is conscious?
Consciousness is a documented phenomenon that has existed since before I existed. My lived experience matches documented evidence of consciousness, thus I can conclude that, by and large, humans are conscious, although I cannot determine directly whether or not another person is conscious without depending on their testament. They could be a philosophical zombie - a thing that has all outward behaviors of a conscious being without having a consciousness, including lying about having a conscious experience - but the likelihood is low.
In the second scenario, there is no “native speaker” that is telling you what to write. There is a list of instructions that you are following that is the same as the instructions that the computer was following with the first scenario.
The argument is that if, by following the exact instructions that the computer is following, you still do not understand the Chinese conversation, why would you say that the machine understands?
I haven’t thought about the idea enough to say I agree or disagree, but I hope I understood your comment and that the explanation helps.
I’d like to add a rebuttal I tend to agree with: You’re only a component of a larger system, and the whole system (you + the room + your list of instructions) does encode some understanding, otherwise a chinese speaker wouldn’t be able to interact with the room in any meaningful way.
In almost any system, you can look at a specific part and argue that part doesn’t fulfill a property that the entire system taken as one does.
But in this case you is unnecessary if a mechanism can perform that part, and understanding then completely doesn’t rely on consciousness. This is a bit weird way to think about it, but it seems to be a semantic difference
No, you’re still a necessary part of the system, since otherwise there is no interaction possible. And yes, I don’t think understanding relies on consciousness.
Where is the semantic difference?
Difference in if that is required and if ‘understanding’ is a quality that a system can have. For instance I feel that a gravity coin sorter is made with understanding but does not have understanding
It’s not really possible to give a concrete answer without a concrete definition of “understanding”. But unless you assume supernatural explanations, either systems must be able to “understand”, or humans are also incapable of “understanding”, since we ourselves are physical systems too.
I get the example and agree that such a coin sorter does not meaningfully have an understanding of gravity. But IMO it does have a (minimal) understanding of the different coins you’re sorting, since it contains a mechanism for differentiating between their sizes.
Imagine we replaced the gravity mechanism with a bunch of actuators controlled by organic neurons, which are connected to something like an eye. The neurons fire and cause the actuators to sort the coins. Now, if say the neurons just react to brightness levels it’d be hard to argue there’s an understanding. But if the neurons are part of a real human brain, they can definitely understand the coins etc. and sort based on their understanding.
As you take away more parts of said brain, when does it lose the understanding?
It’s still the same argument. If the conversation is carried on your end by a bunch of if-else instructions, there’s no consciousness to be found.
Yes, I think it proves that understanding does not require consciousness.
So you’re saying that my program that does
if obj isinstance cat: obj.pet(), understands what a cat is?No, and if you want to discuss the philosophical idea, please don’t waste my time with dumb questions.
No, it doesn’t make any sense at all. If I couldn’t understand when a native speaker was replying, how does me not understanding when a neural network does it make it any less significant?
In the Chinese Room scenario, cards with Chinese written on them are inserted into a box.
Inside there is a man who doesn’t understand Chinese but has a lookup list of which symbols to write as a response to every possible sentence you can write on the inserted card.
For anyone observing from outside the box it looks as if whatever is in the box is fluent in Chinese and can hold a conversation. But the man inside does not understand anything about Chinese.
This is used as an analogy to explain that machines, very much like the man in the box, do not actually understand the data they are fed, nor the data they output, no matter how human the response might seem.
It was an argument against the Turing test, which is about whether machines showing behaviour indistinguishable from a human is a sign that they can think.
The Chinese Room argument is that they must also show understanding of the data itself, rather than just human-like responses, to say that they think.
That’s not what I understood from the Wikipedia article. In any case, an LLM doesn’t look up responses from a table.
An LLM is a series of immutable nodes that pass results on to the next series of nodes until an output is generated, a functionality that could be perfectly imitated with a very large number of people seeing a series of values passed to them from other people, doing a bit of rigidly defined basic math with them, and passing it on to the next set of people who play the part of the next set of nodes.
The fact that these nodes are immutable, have no persistent state, and only process information in one direction (input to output, and no more) means that an LLM has no persistence of experience. You could potentially argue that the information, as it exists, is a form of fleeting consciousness, but it would be so incredibly alien compared to the lived experience of humans that it would be impossible to relate to in any meaningful way - and it would also mean that every time an LLM is run, the consciousness is created and destroyed for every single word that it outputs… unless you can somehow argue that the consciousness is contained in the set of tokens that gets fed back into the LLM with each word it outputs, which would also mean that any string of text is the equivalent of latent consciousness, which is patently absurd.
Curiously, one of the approaches to the consciousness, and a more Zen-ish one, is that clinging to a persistent ‘self’ is erroneous and it’s better seen as a process rather than an entity. Granted, it still assumes a ‘self’ that is changed by each passing experience.
How can you call it absurd when we have no idea what makes something conscious in the first place?
Because:
Consciousness is a discrete phenomenon that exists, and not all things are conscious. A piece of paper, for example, is not conscious.
A string of text can be used to represent any type of data. It would mean that every piece of data is a consciousness, which - if we assume item 1 to be true - cannot be true, as a piece of paper (which is not conscious) can hold the data.
How did you determine that it’s a discrete phenomenon? If I present you with another person, how do you determine if they’re conscious or not?
It’s a discrete phenomenon because my consciousness is not your consciousness. I do not have two consciousnesses at once. Each consciousness is separate from any others - hence, discrete.
As for how I know another person is conscious?
Consciousness is a documented phenomenon that has existed since before I existed. My lived experience matches documented evidence of consciousness, thus I can conclude that, by and large, humans are conscious, although I cannot determine directly whether or not another person is conscious without depending on their testament. They could be a philosophical zombie - a thing that has all outward behaviors of a conscious being without having a consciousness, including lying about having a conscious experience - but the likelihood is low.