
The Recognition-Consequence Gap
Multimodal systems usually label gestures as isolated acts, but a gesture's meaning shows up in what it does between people. We argue that annotation should follow action and consequence across an interaction.
Researcher · Author · Speaker
I use them to build better things.


Multimodal systems usually label gestures as isolated acts, but a gesture's meaning shows up in what it does between people. We argue that annotation should follow action and consequence across an interaction.

People already use gestures naturally, but product teams lean too heavily on what people say.
I study how people use their bodies to think, communicate, and interact with their social and material world, and I use what I learn to build better products. I've spent more than a decade reading those signals in science classrooms, clinics, and software teams. Today I'm a Senior Product Researcher at Microsoft, where I study how people work with AI tools like Copilot.
More about me →