Research suggests we can “hear” hand gestures even when we can’t see them

We’re listening.
We’re listening.
Image: Reuters/Monika Deupala
We may earn a commission from links on this page.

Here’s an experiment to try the next time you’re on the phone with a colleague or friend: As you’re listening to the other person talk, see if you can guess whether they’re also gesticulating with their hands and how.

There’s a strong chance that your conversation partner’s movements will leave an acoustic signature that you may be able to perceive, says Wim Pouw, a cognitive scientist at the Donders Institute for Brain, Cognition and Behaviour at Radboud University Nijmegen in the Netherlands and the author of a new study on this topic recently published in the journal PNAS.

In previous research, Pouw discovered that upper limb and wrist movements created acoustic imprints on vocalizations. Now, working with a team of researchers at the University of Connecticut, he found people who listened to an audio recording of someone saying “ahh” while making a hand gesture as if they were chopping wood (but also keeping their “ahh” as steady as possible) could accurately replicate the speaker’s chopping gestures, at the right tempo, without any visual references. The synchronicity was spontaneous and flawless.

The research suggests that when people speak with their hands, as is common across cultures and languages, the gestures can communicate a bodily state, even when the person is unseen.

To be sure, this was only one study, and it’s small, involving only 30 listeners as participants and six “vocalizers.” But Pouw and his team say the results of the randomized experiment supports the theory that gesturing is not only a form of visual communication meant to enhance the spoken word, but a biological impulse that enhances speech in its own way.

“In body language research, there’s a lot of emphasis on how gestures may embody a concept, or how I may use my hands in conversation to keep the floor before it’s gets taken over,” Pouw tells Quartz. But gesturing, he says, leads to bodily vibrations and muscular changes that affect the respiratory system and thus the pitch and volume of speech.

“It’s information of another kind,” University of Connecticut psychologist James Dixon, one of the authors of the paper and director of the university’s Center for the Ecological Study of Perception and Action, explained in a university report about the study.

The surprising findings may also hint at how verbal and non-verbal language evolved, says Pouw. “If you have a story that is more aligned with the bio-architecture of the body, then you also have a way into how these two systems might have emerged to coordinate together.” When babies flap their arms randomly as they begin to make sounds, it may be a way of exploring the biological relationship between vocalizations and gesture, he argues. “People who are blind use hand gestures when they speak to people who are also blind, so we know this is not about only visual communication, per se.”

What does this mean for the way we work now?

Unfortunately for those suffering from Zoom fatigue, the findings don’t necessarily suggest you can take a break from video chats, switch to a voice call, and still hope to capture all of the richness of hand signals. Pressed on that tantalizing possibility, Pouw said he’d be reluctant to go that far. While his research suggests that something remains of body language when we can’t see it, he also suspects there’s an amplified effect when you see and “hear” hand and body motions.

If anything, he would encourage people to be more expressive with their hands and body, without hesitation, on video calls or the phone. Too often, he says, he sees only floating faces on Zoom, which makes an artificial setting feel even less natural. On video calls, we have to try harder to read what’s happening with other people in the room, something that’s possible without much conscious thought when we’re sharing physical space. So step back, Pouw says, and let your fellow video callers see all of your movement.

And when we’re back at the office?

As the Covid-19 crisis continues, it will be even more imperative to speak with our hands, Pouw advises, citing a recent university interview with one of his colleagues, Asli Özyürek, a cognitive scientist and professor at the Center for Language Studies at Radboud University.

Wearing masks and standing six feet away from others makes it difficult to communicate clearly with our voices alone, as Özyürek points out. And it’s not only the deaf or the hard of hearing who may struggle to understand someone speaking through a mask. Any physical barrier, including clear masks meant to enable lip-reading, will muffle sound.

Fortunately, Özyürek’s research shows we are naturally adept at finding commonly understood hand signals. We know that “when people are asked to communicate only with their hands, speakers of different languages can invent, on the fly, ways to convey complex messages that are easily understood,” she reports.

As we begin to navigate a new corporate world in which desks are set oceans apart, or make-shift plexiglass sneeze guards stand between co-workers, we may see some inventive new gestures yet.