To dig into it further, our team at the Cambridge IDEO studio snagged a camera like the one in the new iPhone and performed a series of experiments to figure out when gesture might be the best choice.

First, we gave pairs of people an idea, then we asked them to make a four-handed pose to express that idea.

IDEO designers pair up to express ideas through gestures.
IDEO designers pair up to express ideas through gestures.
Image: IDEO

Then we recorded stories and tracked people’s hands using computer vision to study when we naturally deploy gestures to amplify emotion or explain a concept.

Ask a group of people to perform an action or request and you get some variation. The trick is finding gestures that are as universal as possible.

Lastly, we trained a neural network to recognize a small set of gestures, and used these to control a Philips HUE light set and a Spotify station to create an installation for the office.

In messing around with these exercises, we discovered that gestures need to be either sequential, like a sentence: noun then verb, object, plus operation. For example, for “speaker, on,” one hand designates the noun, and the other the verb: You point to speaker with left hand and turn the volume up by raising right hand.

(Another surprising insight: Gestures are generation-specific. When asked to signal turning up the volume, a few people twisted an invisible knob, but most of the under 30s lifted a palm or made a pinching gesture with their fingers.)

After analyzing our results, we boiled our thoughts down to four reasons to opt for gesture over voice or touch:

It’s exciting to imagine a whole new category of products that will take advantage of gesture’s subtlety, expressiveness, and speed.

📬 Sign up for the Daily Brief

Our free, fast, and fun briefing on the global economy, delivered every weekday morning.