Eye movement, visual memory, and peanut butter sandwiches

Experiments on change blindness have revealed striking limitations in visual memory. Take a look at the video below, for example (click to play -- and note that the video is contained in a java applet that may take a while to load -- but it's worth it!).

The woman is giving directions to one "construction worker," when two other "workers" carry a door in between them. A new worker (all of them are actually actors hired by the experimenters) is substituted during the hubbub, and the woman continues giving directions as if no change has occurred. This work has been duplicated in a variety of different environments.

How is it that visual memory is so inefficient? If we can't remember the person we were talking to 2 seconds ago, how can we even function in the world?

Recently, vision researchers have begun to take a different approach to studying visual memory. Instead of looking at isolated events, out of context, they are analyzing real-world activities, with real goals.

A team led by Mary M. Hayhoe performed such a study on volunteers as they made peanut butter and jelly sandwiches. The participants wore headgear that simultaneously tracked the movement of their eyes and videotaped the scene before them. After the task was completed, the eye-tracking data was mapped onto the video, and the precise movements of the eye could be tracked to a precision of 1/30 of a second.

Here's an example of one such video (click on the image to play):

The small white circle is an indication of the focus of visual attention. Believe it or not, this small area is the only part of your field of view that you can have in precise focus at any time: it corresponds to the region called the fovea inside your eye: a tiny field with a much higher density of cones -- vision receptor cells -- than the rest of the eye. You build mental pictures of scenes through saccades -- quick eye movements between areas of visual interest.

At the beginning this video, the sandwich-maker's eye is fixated on the knife as she sets it down on the plate. But before she's even let go of the knife, she saccades first to the jelly jar, then to the lid. As soon as she grasps the lid, she saccades again to the jar, where she sustains fixation until she's successfully screwed the lid back on.

Now take a look at this movie, which shows how the participant first explores the sandwich-making apparatus:

As soon as the bread is revealed, she quickly fixates on it. Then her gaze shifts to the plate, the knife, the peanut butter, and the jelly. Once the dropcloth is out of the way, she fixates again on the bread in preparation for the task ahead.

Hayhoe et al. analyzed fixations on a frame-by-frame basis, and found that many fixations lasted less than 100 milliseconds. Take a look at this chart of one participant's fixation durations:

A large portion of the fixations last less than 200 milliseconds, with some as short as 60 milliseconds. This is surprising because several studies have previously found that simply preparing the eye to saccade takes 200 to 250 milliseconds. If many fixations last less than half that time, it suggests that saccades must often be planned in advance -- even several saccades at a time. In the second video above, you can see that the left hand moves to pick up the jelly lid nearly a second before the eye actually saccades to the lid. In the meantime, the eye has fixated on the plate, then the jelly, suggesting that saccades may be planned well in advance.

If this is true, Hayhoe and her colleages argue, then perhaps visual memory isn't as limited as once was thought. In order to plan several saccades in advance, visual memory must maintain a detailed representation of the location of the relevant fixation points.

It is possible that the objects aren't actually maintained in visual memory. Perhaps the rough focused image of the scene in front of the person serves the function of memory; perhaps we don't need accurate visual memory because the very scene we might need to remember is right in front of our eyes. While they acknowledge this may be partially true, Hayhoe's team points to other research showing that objects are more easily found when they have been fixated a few seconds earlier. This is consistent with the evidence shown in the third video above, where the participant first locates each of the objects she will need when making the sandwich.

So though change blindness represents an important limitation of visual memory, Hayhoe et al.'s study demonstrates that the visual system does appear to retain key information about a scene, when that information is relevant to the task at hand.

Apparently, a peanut butter sandwich is a powerful motivator for visual memories!

Hayhoe, M.M., Shrivastava, A., Mruczek, R., & Pelz, J.B. (2003). Visual memory and motor planning in a natural task. Journal of Vision, 3, 49-63.