An evaluation of Searle’s Chinese room experiment

I. Introduction

In his article ‘Minds, Brains, and Programs’ (Searle 1980),1 John Searle argues two things. First, he claims that emulating the functional behaviour of the brain, or some part of it, is insufficient grounds for attributing a machine or computing device with cognitive states such as those experienced by conscious beings like ourselves. Second, that no such device could ever possess such states solely as a result of having the appropriate formal properties, or to put it another way, simply by ‘running the right program’. It is my aim in this essay to show that although Searle’s first claim is correct and successfully refutes the position of ‘strong AI’ (artificial intelligence) as he defines it, the second is not supported by his argument, and so must be considered false, or at best undecided. I will begin by summarising Searle’s argument and what it purports to show before moving on to evaluate it with particular reference to the ‘systems reply’, as described by Block (1995).

II. Searle’s Account

The thought experiment that Searle proposes is effectively a limited version of the test that Turing proposed as a meaningful alternative to the question of whether computers can think (Turing 1950). In it, an English speaking man is shut in a room into and out of which messages written in Chinese may be passed. Although the man has no knowledge of Chinese, he is furnished with a set of instructions or ‘program’, written in English, that describes how to cross reference the incoming symbols to various other sets of Chinese symbols which have been provided to him in order to produce a response, which is then passed back out of the room. Although the man does not know it, the resulting symbols are actually answers to questions relating to a story, written in Chinese, that he was previously provided with. By simply following the program, he is able to respond to the incoming ‘questions’ such that a native Chinese speaker would assume the room to contain another native speaker who actually understands and can respond appropriately to the questions being asked. But since the man in the room has no knowledge of Chinese whatsoever, and is simply following instructions, this raises the question of whether any real ‘understanding’ can be said to have taken place. Does the program somehow enable the man, or in some sense the system as a whole, to understand Chinese? Or is the whole setup merely a simulation that appears to understand the questions being posed, but lacks any real understanding of Chinese, or anything else for that matter, except for the man’s own understanding of the English instructions that he is following?

Searle’s presents his argument as a refutation of a position that he calls ‘strong AI’ (Searle: 353). This position is characterised by the assumption that because a system—in this case a human being following instructions, but a suitably programmed computer will do just as well—appears to exhibit understanding (belief, emotion, and so on), then it must necessarily possess those qualities in exactly the same sense that we do. In other words, exhibiting a particular sort of behaviour is all that is required for the possession of understanding, regardless of how this behaviour is being produced. This is a very strong—some might say naïve—claim, and one that few in the AI community would actually subscribe to, for reasons explained below. Even Newell (1963), who Searle cites as an advocate of ‘strong AI’, talks in terms of describing or simulating human behaviour rather than actually identifying it with a particular computational process.2 However, leaving aside this objection, Searle argues that this claim is simply untrue. It takes more than just behaviour to ascertain the existence of cognitive states, or intentionality. The behaviour must be generated by particular kinds of processes occurring, according to Searle, in the right kind of physical stuff—i.e. the brain—in order to qualify as truly intentional (Searle: 365). Simply following a set of instructions is not a sufficient or necessary condition for intentionality.

So far, at least, Searle’s account seems relatively uncontentious. We would not ascribe intentionality to a system that we knew to be devoid of symbolic thought. This becomes obvious when we consider the simplest possible implementation of the Chinese ‘understanding-simulator,’ which would consist of a simple list of all the answers to all the questions that could possibly be asked about all the possible stories that could be submitted. Of course, this would be a very long list, but in the spirit of the Turing machine and other thought experiments, we will grant the subject such an exhaustive list and the ability to simply read off the correct answer to each question from the list as it arrives. In this case, it is clear that no real understanding takes place, even though the functional requirements of producing the correct answer to each question are satisfied as before. We can therefore conclude that satisfying functional requirements is not a sufficient condition for understanding. This confirms what I shall call Searle’s weak thesis, which suggests that understanding requires something over and above mere behaviour or function. As this conclusion seems beyond contention, I shall not consider it any further in this essay. However, Searle goes on to claim that no purely formal system, i.e. any system defined in terms of formal symbols and processes, is capable of exhibiting intentionality by definition. In other words, no computer, either now or in the future, could generate true understanding (belief, emotion, etc.) as a result of running any conceivable program—it simply isn’t possible. Although Searle explicitly grants that computers may one day be able to think (Searle: 368), he claims that mental processes such as understanding cannot be the result of ‘computational processes over formally defined elements’ (Searle: 367). This is what I will call Searle’s strong thesis, and is a claim that must be examined in some detail.

III. The Systems Reply

Searle’s strong thesis hinges on the claim that if the Chinese room experiment was to involve any real understanding of Chinese, then this understanding would be available to the man who is following the program. Therefore if the man does not understand Chinese as a result of executing the program, then no understanding has taken place (Searle: 356; 359). However, it is far from clear that this should be the case. Firstly, the man is simply one component of the whole system, and not the system itself. The man in the Chinese room experiment is analogous to the central processing unit (CPU) of a conventional computer system. We normally attribute behaviours such as running, jumping, talking, or seeing to a whole person rather than just their brain, which is after all, the CPU of the body. In the same way, any understanding that arises as a result of the Chinese room experiment would reside in the system as a whole and not any one component of it. This is what Searle terms the systems reply to his argument (Searle: 358). His response is to posit a modified version of the experiment in which the man internalises the various components of the system—symbols, instructions, and so on—so that the entire process takes place within the confines of his own head. Here, exactly the same program is being followed with identical results: the system appears to ‘understand’ Chinese, but the man himself does not, even though the former is no longer physically distinct from the latter. Searle takes this to prove his point that no understanding takes place, but as Block (1995) points out, the argument is invalid. Just because the man implements the Chinese room system does not give us any reason to identify him with it. Even though the details of the Chinese room program are now hidden from view, the system still exists as a distinct entity within the man, and so there is no reason for us to suppose that its properties should be directly attributable to him any more than his properties are attributable to it. Just because his conscious thought processes form the mechanism by which the Chinese room program operates does not mean that he would have conscious access to any understanding, thoughts or other higher-level phenomena taking place within the system. The two are operating at entirely different levels of abstraction and would be completely oblivious to each other’s ‘thought processes’, assuming the program has such things. Even if their internal symbolic representations were similar enough that they could understand one another, there is no way that this information could ‘jump’ across the two levels. Searle’s argument is invalid because there is no reason to suppose that any understanding generated by the Chinese room system would be accessible to the man, and so the fact that the man gains no understanding of Chinese by following the program does not show that the system itself has no understanding. Short of testing the system’s behaviour by asking it questions, or making empirical observations about its internal state, the matter of whether the system has understanding remains an open question, and Searle’s argument fails to prove anything one way or the other.

IV. Other Issues

Although this point is sufficient to invalidate Searle’s strong thesis, there are several other difficulties with his argument that warrant further attention. Searle argues that any system that merely manipulates syntax, i.e. formal symbols, is incapable of producing semantics, i.e. meaning (Searle: 370). However, if a symbol’s meaning is understood as the role it plays within a complex network of interrelated symbols—the so-called network theory of meaning (Churchland 1988: 56)—or in relation to some particular ‘structured context’ (Gewirth 1982: 108), then it is possible that understanding is simply a matter of knowing the relationships between various sets of syntactic symbols. Searle goes on to suggest that the brain has particular ‘causal powers’ (Searle: 367) that are required for intentionality, but fails to specify what these are, or why they are so important.3 Without placing these claims in the context of a particular theory of meaning, or an explanation as to what differentiates physical brain processes from mere formal symbol manipulation, it is difficult to assign much value to them. Searle is also guilty of somewhat misrepresenting the Turing test, which is intended to provide a well-defined alternative to the question of intelligence, and not an exact equivalent of it.4 That a program is actually ‘a specification for a machine’ (Sharvy 1985), rather than a set of instructions that the machine literally follows, as Searle assumes (Searle: 372), causes the Chinese room analogy to break down when we consider that the man in the experiment is explicitly following instructions, whereas a machine can merely be described as such. In fact, a computer is simply a machine that has been prepared to produce the required behaviour, making it conceptually very similar to the brain, which could be regarded as a machine for producing intentionality. This point is particularly relevant when we consider the nature of neural networks or parallel distributed processing (PDP) systems, which consist of a large number of incredibly simple formal systems that may be programmed or ‘trained’ to perform highly complex and sophisticated tasks. These function in much the same way as neural circuits in the brain, which act in concert to produce what we regard as intelligent thought and behaviour. Although such massively parallel systems could in theory be simulated by any Turing machine, such as a man sitting in a room manipulating a set of symbols, the properties of the two are so vastly different as to make any intuitive judgements that we may reach about what such a system is capable of virtually meaningless.5

V. Conclusion

In summary, Searle’s argument is successful in refuting the (hypothetical) claim of ‘strong AI’ that he sets himself. However, his weak thesis only proves that apparently intentional behaviour is insufficient grounds for attributing the sort of mental states that are required for intentionality; it does not prove that such states cannot be generated by formal symbol manipulation systems, such as computers. Searle’s strong thesis fails to prove the latter point as it conflates the intentional properties of the system (the Chinese room) with those of the hardware used to implement it (i.e. the CPU or host). In the absence of any further explanation of what is meant by ‘understanding’ or the ‘causal powers’ of biological systems, the possibility that an appropriately programmed or trained computer could exhibit true intentional states, such as understanding or even consciousness, is left open. Whether such an entity would sustain the same kinds of thoughts, experiences and sensations as we ourselves enjoy, however, is open to question, and remains a legitimate matter for debate amongst the AI and philosophical community.

——————

Bibliography

Block, Ned 1995: ‘The Mind as the Software of the Brain’. In An Invitation to Cognitive Science, Osherson, Gleitman, Kosslyn, Smith and Sternberg (eds.), vol. III,
ch. 11.

Churchland, Paul 1988: Matter and Consciousness (Revised Edition). Cambridge, Massachusetts: MIT Press.

Dennett, Daniel 1981: Mindstorms. London: Penguin.

Gewirth, Alan 1982: Human Rights: Essays on Justification and Applications. Chicago: University of Chicago Press.

Hofstadter, Douglas and Dennett, Daniel (eds.) 1981: The Mind’s I. Brighton: Harvester Press.

Newell, Allen and Simon, H A 1963: ‘GPS, A Program That Simulates Human Thought’. In Computers and Thought, Feigenbaum and Feldman (eds.),
pp. 279–293.

Searle, John 1980: ‘Minds, Brains, and Programs’. In The Mind’s I, Hofstadter and Dennett (eds.), pp. 353–73.

———— 1992: The Rediscovery of the Mind. Cambridge, Massachusetts: MIT Press

Sharvy, Alan 1985: ‘Searle on Programs and Intentionality’. Canadian Journal of Philosophy, supplementary vol. 11, pp. 39–54.

Turing, Alan 1950: ‘Computing Machinery and Intelligence’. Mind, 59, pp. 433–460.