Sound and vision work hand in hand, UCLA psychologists report

Ladan Shams

"If we think of the perceptual system as a democracy where each sense is like a person casting a vote and all votes are counted to reach a decision — although not all votes are counted equally — what our study shows is that the voters talk to one another and influence one another even before each casts a vote," said Ladan Shams, a UCLA associate professor of psychology and the senior author of the new study.

In the study of how one sense can affect another, Shams, an expert on perception and cognitive neuroscience, and her colleagues showed 63 participants a large number of dots on a screen in two separate phases, with a break between the phases. In one phase, the dots moved around randomly; in the other, some of the dots moved together from right to left. In both phases, the dots were accompanied by sound.

Over a series of experiments, the researchers asked the participants to correctly identify the phase in which the dots moved together horizontally.

Participants were divided into three groups. One group heard sound moving from right to left as the dots moved from right to left, and they heard sound that remained stationary during the random phase. A second group heard the right-to-left sound during both phases. And a third group heard sound moving in the opposite direction — from left to right — during both phases. Then, each participant experienced trials in all three conditions.

As Shams expected, the participants were best able to identify the phase in which the dots moved horizontally when the sound moved in the same direction as the dots but remained stationary during the random phase. The researchers found that the sound that moved in the opposite direction neither enhanced nor worsened the participants’ visual perception.

Surprisingly, the sound that traveled leftward both when the dots moved leftward and when the dots moved randomly — that is, sound that provided no useful information for choosing between the two phases — also helped people correctly choose the phase with the horizontal motion. Because the sound was identical in both phases, if the participants closed their eyes they, would have a 50-50 chance of successfully performing the task based on sound alone; with their eyes open, however, the interaction between sound and vision led to a significant improvement in detection of visual motion. Hearing enhanced seeing.

"Imagine you are playing ping-pong with a friend who serves the ball," Shams said. "You receive information about where and when the ball hit the table by both vision and hearing. Scientists have believed that each of the senses produces an estimate relevant for the task, and then these votes get combined subconsciously according to rules that take into account which sense is more reliable. This is how the senses interact in how we perceive the world. However, our findings show that the senses of hearing and vision can also interact at a more basic level, before they each even produce an estimate."

The senses affect one another other in many ways, said Robyn Kim, a former UCLA psychology postdoctoral scholar in Shams’ laboratory and lead author of the study. There are connections between the auditory and visual portions of the brain at the cognitive level. When the information from one sense is ambiguous, another sense can step in and clarify or ratify the perception, Shams and Kim said.

"This study shows that at least in regards to perception of moving objects, hearing and sight are deeply intertwined, to the degree that even when sound is completely irrelevant to the task, it still influences the way we see the world," Shams said.