Future of gesture and sign recognition, research, and applications - VaIL Discussion Starter

1 view

Skip to first unread message

sunny

unread,

Jul 31, 2007, 4:48:28 PM7/31/07

to Visual and Iconic Language

Topics to be discussed at VaIL 2007:

* Area overview, what has been done, what needs to be done
* Inscription technology (gloves, motion capture, ...)
* Sign recognition vs gesture recognition (differences in approaches
and requirements)
* Near term applications (Wii/games, information entry, data
manipulation, Sign translation, Sign and gesture annotation/tagging)
* Capturing and translating iconicity (iconic gesture/sign to iconic
language)
* Applying automated linguistics techniques to Sign part-of-speech
tagging
* Methods for gesture recognition
* Gesture and sign annotation

Please help us flush out answers to these questions and to find the
right questions.

!! Respond with your thoughts and ideas !!

Jack Ox

unread,

Aug 6, 2007, 10:58:29 AM8/6/07

to visual-...@googlegroups.com

Hello all- did you see this?

washingtonpost.com

ad_icon

Gestures Convey Message: Learning in Progress

By Rick Weiss

Washington Post Staff Writer

Monday, August 6, 2007; A06

Susan Wagner Cook stands at the front of a third-grade classroom, an unfinished equation printed neatly on the whiteboard.

4 + 3 + 6 = __ + 6

"I want to make one side," she says, as her left hand sweeps under the left side of the equation, "equal to the other side," she continues, now sweeping her right hand under the right side of the equation.

It's a concept that third-graders are just ready to learn: The total value on one side of an equal sign should equal that on the other.

Some kids get it quickly as Cook goes through her carefully choreographed tutorial. Others take longer. But what none of them know is that they are subjects in an experiment that is helping scientists understand one of the most familiar and yet mysterious components of human behavior: the hand gesture.

Teachers who use gestures as they explain a concept -- such as the hand sweeps that Cook uses to emphasize an equation's symmetry -- are more successful at getting their ideas across, research has shown. And students who spontaneously gesture as they work through new ideas tend to remember them longer than those who do not move their hands.

Now Cook's work with elementary schoolchildren is helping to find out whether the gesturing done spontaneously by many quick learners is simply a reflection of the fact that they are "getting it" or is actively helping them learn.

Her findings, along with others in the emerging field, could open new vistas in neuroscience, cognitive psychology and education. They may even bring a modicum of science to such pressing questions as: What is it with those Italians?

"Everyone gestures," said Cook, a postdoctoral student at the University of Rochester, deferring at first on the Italian question. "People start gesturing before they can talk, and they keep gesturing for their entire lives."

Even blind people gesture when they talk, as do people chatting on telephones -- proof that gesturing is not necessarily for the person who is listening. In many cases, it seems, gesturing has nothing at all to do with communication.

But then, what is its purpose?

Before trying to answer that question, experts say, think beyond the gestures that might spring to mind -- the extended middle finger of the aggressive driver or the athlete's pumped fist, the meanings of which can be read as plainly as words on a page.

Rather, think about the way your arms flutter as you try to describe something almost ineffable. Or the way your hand wags up and down as you struggle for a word, as though you might physically retrieve it from some sticky mental crevasse.

These are the kinds of gestures that offer a window on the murky link between body and mind, and which in recent years have given rise to an International Society for Gesture Studies, a scientific journal (aptly named Gesture) and a newsletter called Manufacts.

"I've really been struck by how sophisticated and focused the field has become," said David McNeil, a professor emeritus of psychology and linguistics at the University of Chicago, the hotbed of gesture studies where Cook did her seminal work on the educational value of gestures. "It's really gaining momentum very rapidly."

The new focus on gestures comes at a time when researchers have been moving away from the old model of the brain as a sophisticated computer -- a model that came to prominence in the 1950s, when computers seemed to hold the answers to everything.

"People have started to realize there's something wrong with this model," said Arthur Glenberg, a gesture researcher at the University of Wisconsin in Madison. "We're not just dealing with zeros and ones. We're biological beings, and we ought to consider how we deal with the real world and take seriously the fact that we have bodies."

Neuroscientists have found, for example, that the part of the brain that controls hand movements is often active when people are doing math problems. "As though you're counting fingers," Glenberg said.

Similarly, parts of the brain responsible for speech are often active when people gesture -- more evidence of the link between language and movement, aside from formal sign language.

Observations such as these have led to a number of experiments to test the idea that gestures might help with memory and learning.

Glenberg has shown that when elementary and middle school students are asked to move objects about as they read a story -- placing a toy farmer on a toy tractor after reading a phrase about that activity, for example -- they scored better on tests about what happened in those stories.

Cook's latest work goes further, showing that even abstract gestures can enhance learning. In a classroom, she had some students mimic her sweeping hand motions to emphasize that both sides of an equation must be equal. Other students were simply told to repeat her words: "I want to make one side . . . equal to the other side."

A third group was told to mimic both her movements and words.

Weeks later, the students were quizzed. Those in the two groups that were taught the gestures were three times as likely to solve the equations correctly than were those who had learned only the verbal instructions, she and two colleagues reported in the July 25 issue of the journal Cognition.

Previous studies showed only that students who spontaneously gesture learn better, leaving uncertain whether gestures simply reflect an emergent understanding or help create it. The finding that even concocted gestures improve scores "allows us to make the scientific argument that gesturing is causing an improvement in learning," Cook said.

In effect, she concluded in a report last year with the University of Chicago's Susan Goldin-Meadow, "Children may be able to use their hands to change their minds."

How gestures help consolidate new information remains a mystery. One idea is that gestures help a listener focus on what is being said. But experiments by McNeil -- in which a teacher makes gestures that don't match what is being said -- suggest that gestures have effects on their own, independent of speech.

Another possibility is that gestures lighten the cognitive load during problem solving, leaving more mental power for memory. In one study, people were asked to memorize a list of words and also explain a math problem. Those who were allowed to gesture while explaining the math did better on the memorization than those who were told not to move their hands.

Gesture and speech may also work best on different kinds of information.

"The meaning of speech is clear, but in some cases it may be too precise," Cook said. "Gestures may be better at conveying broader concepts."

So what about those Italians?

As it turns out, they probably don't deserve their reputation for gesturing more.

"The biggest difference between cultures seems to be the size and space you're allowed to take up with your gestures," Cook said.

Italians, she said, simply "gesture larger."