Action and Meaning

Throughout the brief history of cognitive science, debates over the nature of knowledge representation have raged. In the 1970s, the debate was between those who thought that knowledge was represented as images -- modal, or sensory representations -- and those who thought that knowledge was represented propositionally. That particular debate ended in a stalemate, upon the realization that you could account for pretty much any data set from either perspective. If you can't distinguish between perspectives, you can't really debate them. Despite the stalemate, most cognitive scientists who've studied knowledge representation over the last two decades have adopted the propositional perspective.

However, with the publication of a paper titled "Perceptual Symbol Systems" by Larry Barsalou in 1999, the imagists began to make a comeback under the guise of perceptual symbol systems (PSS) theory. The basic idea behind PSS is that all knowledge representations receive their content from perceptual information. For example, when we activate our represenation of a zebra, we are activating an image of zebras. Reasoning then occurs through perceptual simulations, which I usually describe as little movies in the head (though keep in mind that perceptual representations need not be visual -- they could be auditory, gustatory, olfactory, haptic, or whatever).

The motivation for PSS is as old as thinking about thinking: the symbol grounding problem. PSS theorists argue that it impossible to connect amodal (i.e., non-perceptional, arbitrary symbols, like words) to the perceptual content that we get from the outside world. The problem is often expressed in versions of Searle's famous Chinese Room thought experiment. For example, Steve Harnad gave us " The Chinese/Chinese Dictionary-Go-Round," which goes like this:

My own example of the symbol grounding problem has two versions, one difficult, and one, I think, impossible. The difficult version is: Suppose you had to learn Chinese as a second language and the only source of information you had was a Chinese/Chinese dictionary. The trip through the dictionary would amount to a merry-go-round, passing endlessly from one meaningless symbol or symbol-string (the definientes) to another (the definienda), never coming to a halt on what anything meant.

The only reason cryptologists of ancient languages and secret codes seem to be able to successfully accomplish something very like this is that their efforts are grounded in a first language and in real world experience and knowledge. The second variant of the Dictionary-Go-Round, however, goes far beyond the conceivable resources of cryptology: Suppose you had to learn Chinese as a first language and the only source of information you had was a Chinese/Chinese dictionary! This is more like the actual task faced by a purely symbolic model of the mind: How can you ever get off the symbol/symbol merry-go-round? How is symbol meaning to be grounded in something other than just more meaningless symbols? This is the symbol grounding problem.

The problem, as expressed by Harnad and many PSS theorists, is that in order to be able to interpret and use a symbol, you'll need another symbol that expresses its meaning, and so on, in a circle or an infinite pile of symbols.

i-60bcdfa0ed8606a94e198a997b7852af-PSS.gif

The incredibly sophisticated solution to the symbol grounding problem in PSS theory: lines.1

The symbol grounding problem is an important one, but as soon as Barsalou published his paper on PSS, it became clear to most researchers that it wouldn't solve it. The biggest fault of PSS is that to make it work, you ultimately need some amodal representations (as Barsalou and others later admitted2). If you need amodal representations, then you either have to admit that amodal representations can be connected to perceptual representations, or that PSS suffers from the exact problem that strictly symbolic (propositional) models do. Furthermore, since PSS is essentially just another version of the mental image perspective from the 1970s, it suffers from the same problem that that perspective did: how do you distinguish it, empirically, from propositional accounts? It turns out that you really can't. So, PSS doesn't really get us anywhere theoretically, and for the most part, most cognitive scientists still think of representations in propositional terms. But PSS has done one thing for the discipline: produced a bunch of interesting experiments. In the rest of this post, I'm going to describe one set of experiments, to give you an example.

One version of PSS is Art Glenberg's3 indexical hypothesis of language comprehension. I'll let Glenberg explain the basics of the indexical hypothesis to you4.

One such account is provided by the indexical hypothesis (IH), which proposes that meaning is based on action. For example, consider how a situation (e.g., a room with a chair) could be meaningful to an animal. By hypothesis, the meaning of the situation consists of the set of actions available to the animal in the situation. The set of actions results from meshing (i.e., smoothly integrating) affordances to accomplish action-based goals. Affordances are potential interactions between bodies and objects. Thus, a chair affords sitting for adult humans, but not for mice or elephants, who have the wrong sorts of bodies to sit in an ordinary chair. A chair also affords standing-on for the human. If the human has the goal of changing a light bulb in a ceiling fixture, the meaning of the situation arises from meshing the affordances of a light bulb (it can be held in the hand) with the affordances of the chair (it can be stood on to raise the body) to accomplish the goal of changing the bulb. (pp. 558-559)

The indexical hypothesis involves three processes: mapping words, phrases, or sentences onto perceptual representations (you know, mapping amodal symbols onto perception -- the very problem PSS is supposed to avoid), then deriving affordances from the perceptual symbols, and finally transforming the words, phrases, or sentences into "action-based meanings." For example5:

[O]ne can judge that the sentence, "Hang the coat on the upright vacuum cleaner" is sensible, because one can derive from the perceptual symbol of the vacuum cleaner the affordances that allow it to be used as a coat rack. Similarly, one can judge that the sentence "Hang the coat on the upright cup" is not sensible in most contexts, because cups do not usually have the proper affordances to serve as coat racks. (p. 569)

As a test/demonstration of the indexical hypothesis, Glenberg and Kaschak conducted a study in which participants read sentences one at a time, some of which made sense and some of which did not (e.g., "Boil the
air."), and were told to indicate as quickly as they could whether each sentence made sense. Of the sentences that did make sense, half involved movement away from the body (e.g., "Close the drawer.") and half motion towards the body (e.g., "Open the drawer."). In order to respond to each sentence, participants had to press one of three buttons, one close to the body, one about arms length from the body, and one in between the two. In the away-from-the-body condition, responding that the sentence made sense involved moving the hand from the middle button to the furthest button (and thus away body), and in the towards-the-body condition, responding that the sentence made sense involved moving the hand from the middle button to the closest button (and thus towards the body). The indexical hypothesis predicts that participants will be able to respond faster when their movement is consistent with the movement in the sentence. When participants read the sentence "Open the drawer," for example, they will respond that it makes sense faster when they answering involves moving from the middle to the closest button.

Guess what? It worked. Participants' reaction times improved between 25 and 100 ms, depending on the sentence type, when the movement was consistent with the meaning. Glenberg and Kaschak argue that this supports the idea that our representations of the sentences are coupled to representations of action. Presumably, reading the sentence primes the toward or away movement, which allows us to perform it faster. And you know, I buy that explanation. It makes a lot of sense that thinking about a movement would prime that movement. Does that imply that our representation of the sentence is composed of perceptual and motor information? Well, no, but who cares? As long as a few cognitive scientists do believe that, we'll get cool experiments like this one. Sure, the experiment doesn't really tell us anything about sentence comprehension (other than that, you know, sentences have meanings that are connected to the world -- wasn't that the problem we were trying to solve, not to demonstrate? I forget sometimes), but it was fun to read about it.


1From Zwaan, R.A., & Madden, C.J. (2005). Embodied Sentence Comprehension. In: D. Pecher & R.A. Zwaan (Eds.), The Grounding of Cognition: The Role of Perception and Action in Memory, Language, and Thinking. Cambridge, UK: Cambridge University Press.
2E.g., Prinz, J. (2002). Furnishing the Mind. Cambridge, MA: Bradford Books, MIT Press.
3Glenberg, A. M. (1997). What memory is for. Behavioral & Brain Sciences, 20, 1-55.
4Glenberg, A. M., & Kaschak, M. P. (2002). Grounding language in action. Psychonomic Bulletin and Review, 9, 558-565.
5Glenberg & Kaschak (2002).

Categories

More like this

One of the more sophisticated theories in embodied cognition is Lawrence Barsalou's perceptual symbol systems theory. It is, in essence, an updated version of the "ideas as images" position of the British empiricists, and the mental imagery theories of the seventies1. The basics of the theory are…
One of the more sophisticated theories in embodied cognition is Lawrence Barsalou's perceptual symbol systems theory (which I've talked about before, here). Starting from the premise that cognition is for action, it argues that concepts are represented in the same brain systems that directly serve…
In his comment to my post on conceptual metaphor theory (CMT), reposted here, Dr. Gibbs writes: The topic of why conceptual metaphor theory arouses such vehemence is one that greatly interests me and is again the subject of my in progress book. My own opinion is that conceptual metaphor theory, and…
The capacity to use and manipulate symbols has been heralded as a uniquely human capacity (although we know at least a few cases where that seems untrue). The cognitive processes involved in symbol use have proven difficult to understand, perhaps because reductionist scientific methods seem to…

The Glenberg & Kaschak (2002) experiment is really cool, but I agree with you that it does not necessarily rule out other alternatives. While reading about the G & K study, I thought about earlier work on priming and stereotypes, which might be interesting to your readers. I am too lazy to explain the study I had in mind, so I will quote from Bargh, Chen, & Burrows (1996):

"The present experiments were designed to study the effect of activation of the elderly stereotype on behavior. Participants were instructed to work on a scrambled-sentence task as part of a language proficiency experiment. The scrambled-sentence task contained words relevant to the elderly stereotype in the elderly priming condition, but all references to slowness, which is a quality stereotypically associated with elderly people, were excluded. The neutral priming condition scrambled-sentence task contained age-non-specific words in the place of elderly stereotyped words.

After completing the task, each participant was partially debriefed and thanked for his or her participation. A second experimenter then surreptitiously recorded the amount of time the participant took to walk down the corridor after exiting the laboratory room...The main hypothesis was that participants who had been primed with the elderly stereotype would walk more slowly compared to participants who had not been primed with the stereotype-relevant stimuli...

...Participants in the elderly priming condition (M = 8.28 s) had a slower walking speed compared to participants in the neutral priming condition (M = 7.30 s), t (28) = 2.86, p ⤠.01, as predicted."

Neat, huh?

Off-topic: could you post on the Jeffrey Mogil study on empathetic mice which came out in Science this week? It's already gotten some press coverage, and I can't access the actual paper yet.

Ceresina, Barsalou's responsible for one of the classic papers in cognitive psychology ("Ad hoc categories") and since he became influenced by the flickering lights of variance (that's a joke for concept researchers) and thus proposed PSS theory, he's done some really cool work. Rolled up lawns? That's just cool.

But yeah, it can be frustrating.

Mark, I read the paper, and I could probably post on it, but let me ask around ScienceBlogs and see if there's someeone who's more qualified to do so first.

I know it's not apt to be that popular, but C. S. Peirce just took the position way back in the 19th century that one ought just embrace the symbol problem and accept an actual infinity of signs between any two signs. I'd discussed Peirce's solution (with one of his famous mathematical illustrations) several times as well as in connection with Donald Davidson.

The problem for those arguing for consciousness to be purely neurological (which Peirce would not) is that one wants to narrow symbols to states of brain cells or something akin to that. Thus it would appear, at least at a glance, that this is a finite set of signs (broadly conceived) Thus I don't think most would accept Peirce.

Hey Clark, that's one thing that I actually do know about Pierce (finally you mention one I do know), and I thought about that as I was posting yesterday. With a Piercean conception of signs referring to signs, you still have a problem of connecting that to the outside world (even if the outside world is always a series of signs). The symbol grounding problem becomes different from what it is in Searl or Harnad, but it's still a problem... I think.

The solution of PSS isn't that new, either. The indexical hypothesis looks suspiciously like the loop between action and the world in Matter and Memory, for example. The only difference is the mediating role of Gibson's affordances. The problem, of course, is how to connect affordances to images (to say nothing of abstract concepts -- what is the image of "right" or "wrong?").

Chris, I only brought it up because I found out about Michael Tomasello's position here a while ago, and the mouse study seems to provide evidence contrary to his theory.
Clark, if I remember his book correctly, Terrence Deacon seemed to embrace a very Piercean symbol system. It was, for me, one of the hazier parts of 'Symbolic Species'.

Hey Mark, definitely no problem bringing it up. It is interesting, though I'm not sure it's completely contrary to Tomasello's hypothesis. I'll see what I can put together.

I don't think, Chris, that the symbol grounding is a logical problem but you have to buy into Peirce's notion of continuity which is a hypothesis I suspect many might not. The other thing to keep in mind is that Peirce is a holist and an externalist. So the inside/outside dichotomy doesn't necessarily apply.

Yeah, I know about that Barsalou and his history...
(Grumble mumble *Barsalou*)
No, I mean, I do know why he's important, and I understand how that was important to the history of the science, but he still makes me cranky.