Hand Pose Recognition

[Joshua Blake]

Attached is an image showing my progress on hand pose recognition.

This uses my implementation of the Kinect Body Part Recognition MSR paper. It runs faster than real time with less than one CPU core.

The decision forest used to recognize the hands in this image was trained with 625 computer generated and labeled images representing the combination of five images for each of four degrees of freedom for the hand/wrist range of motion. Training three trees in the forest  took just over 11 hours on a Core i7 2600. Additional degrees of freedom are planned, but take longer to train. I am also looking at post processing the classified pixels to reduce joint jitter.

Thanks,
Josh

Technical Director,
InfoStrat Advanced Technology Group

[Murilo Saraiva de Queiroz]

That's amazing! Kudos, Joshua! I'm looking forward to use your implementation. 

Murilo Q. 

--
You received this message because you are subscribed to the Google Groups "OpenNI" group.
To post to this group, send email to openn...@googlegroups.com.
To unsubscribe from this group, send email to openni-dev+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/openni-dev?hl=en.

--

Murilo Saraiva de Queiroz, MSc

Senior Software Engineer

http://www.tecnologiainteligente.com.br
http://www.igenesis.com.br 

http://www.vettalabs.com

http://muriloq.com/blog

[Rofli Sanches]

Nice Blake.

Hard work is worth.

--
Atenciosamente,
Rofli Sanches de Souza.

[Sam Muscroft]

Interesting work Josh.

Could you clarify which hand poses you're classifying? It would also be good to know how many hand poses can be trained and recognized in a training set. I'm also curious whether the accuracy and reliability of recognition deteriorates when the number of gestures in a training set is increased and at greater distances from the sensor. Your diagram suggests that recognition is reliable at variable ranges . 

I've been working on hand posture recognition recently using machine learning and image processing so would be good to discuss this with you. I'm using opencv and SVM , sample collection is done in near real-time and training takes a few seconds to complete. However using this approach, accuracy deteriorates significantly at range from the sensor. This is largely due to the low resolution of the depth map.  This approach allows for the recognition of poses such as the number of fingers on a hand, open / closed hand, rock n roll sign, telephone gesture etc..

Drop me an email if you'd like to discuss further.

Thanks,

Sam.

[Joshua Blake]

Sam,

 

I'm not recognizing a specific hand pose. I'm using the technique described in this paper http://research.microsoft.com/pubs/145347/BodyPartRecognition.pdf to classify individual depth image pixels into labels that represent the part of the body they most likely belong to. A labeled image is that multi-colored suit from my image and the paper. We then analyze the spatial relationship of the labels to determine skeleton joints. This algorithm is used on Xbox for Kinect and in the Kinect for Windows SDK.

 

With sufficient training data and the right training parameters, it can identify the parts of the body (or anything really) in any orientation and pose. From there you would take the skeleton joints and do gesture recognition or whatever you like.

 

The accuracy and reliability does not deteriorate from adding more training images. It gets better and more robust in general. The design of the algorithm is depth invariant so depth doesn't matter at all.

 

Josh

[Sam Muscroft]

Thanks for the info Josh. I'll read the MSR paper. Is it necessary to manually label the body parts in order to identify features?

Sam.

[Joshua Blake]

Thanks for all the comments so far.

 

I've uploaded a video of this work-in-progress:

http://www.youtube.com/watch?v=Nbc1x76L248

 

Still working on training larger sets of training images and increasing the number of skeleton joints reconstructed.

 

Thanks,

Josh
---
Joshua Blake
Microsoft Surface MVP
OpenKinect Community Founder http://openkinect.org

(cell) 703-946-7176
Twitter: http://twitter.com/joshblake
Blog: http://nui.joshland.org
Natural User Interfaces in .NET book: http://bit.ly/NUIbook