Difference between open hand and closed hand

[Jeff de Veth]

I am working on a project in Unity3D using hand tracking with the
Kinect. For this project I am writing my own gesture recognition, but
I have ran into a problem.

I need to be able to tell the difference between a push and a punch
gesture. For the most part these gestures are the same, the important
difference between them is that a push gesture uses an open hand and a
punch gesture uses a closed hand. I have been looking around in the
code, but have had no success in differentiating between the two so
far. It looks like you can only get the position of the hand and no
other information is available. Am I just missing something obvious,
or is it even possible to differentiate between an open and closed
hand?

I'm using
OpenNI version 1.5.2.23
Nite version 1.5.2.21
SensorKinect version 5.0.3.3
Zigfu Unity bindings version 1.4

[Kumar Ahir]

Hi Jeff

I am also working with the same setup trying to detect hand gestures
particularly open and close.
I am successful in recognizing this gestures outside unity using
opencv in c++ environment. However to do so in unity I am thinking of
following approach.
We need a Unity pro license to use this c++ dll as plugin.
Hence I am trying to redo the entire code in c# and then make dll out
of it and use its method in unity. Surprising you can make dll out of
C# code and use it in unity base/indie version.

Given this i am going to do following steps

1. get the frame of hand
2. pass the image frame to a method in c# code. this method will tell
me whether the hand is open or close
3. take appropriate action

I have been doing research on different ways to do step 2. One is as i
mentioned to rewrite my code in c#. Another are Emgu and Aforge C#
libraries that are wrappers of opencv in C# and provides most of the
opencv functionalities.
But but but, being novice in c# i am stuck at step 2.

Regards,
Kumar

[Shlomo Zippel]

Hey Kumar,

Although you need the pro version of unity to write plugins, you can easily call your external code even in the free version.

C# has a nifty feature called P/Invoke - you can use this method to call functions in an external c/c++ dll you write. The easiest way to do this is to expose c-style functions from your DLL:

__declspec(dllexport) bool __stdcall isHandOpen(unsigned char * imageMap, int x, int y)

and then P/Invoke them from C# within unity:

using System.Runtime.InteropServices;

class YourDllWrapper {

    [DllImport("yourdllname")]

    public static extern bool isHandOpen(IntPtr imageMap, int x, int y);

}

And then you can simply call your method:

ImageGenerator Image = OpenNIContext.OpenNode(NodeType.Image) as ImageGenerator;

YourDllWrapper.isHandOpen(Image.ImageMapPtr, hand.x, hand.y); // make sure to convert hand point to projective coordinates, not real world

Of course, as you wrote, you can also use Emgu directly in unity3d and implement your palm detector within unity. If you do that make sure to set the .NET compatibility level to 2.0 in the build settings, otherwise you'll get weird errors! (We have already successfully used Emgu with image map data in unity3d, please let me know if you would like some reference code for this)

Hope that helps,

Shlomo

[Kumar Ahir]

Hi Shlomo,

Please do share the reference code.
With the first method (making dll out of c++ code and calling it from
unity c# through P/Invoke feature) has a barrier of different
datatypes. c++ code is in openframeworks format (ofxCvImage for image
data type) and the image in unity c# is image pointer. Don't know how
to do conversion !

So am thinking of taking the second approach of using emgucv, which
again am novice at.

Your help will be much appreciated.

Thanks,
Kumar

[Kumar Ahir]

Hi Folks,

I am able to get the entire hand detection code running in c#. I checked this by writing a console application and giving hand image as input file and getting an output file with convexity defects.

I am also able to call the functions in c# dll from within c# in unity.

Now am stuck at the gates.

I am retrieving the DepthMap image in unity in c# like

OpenNIContext.Instance.Depth.DepthMapPtr 

and passing to a method in c# dll which has following method definition

public int detectHandPose(IntPtr srcptr,Int32 w,Int32 h)

        {

            Bitmap srcbmp = new Bitmap(w, h, 2, System.Drawing.Imaging.PixelFormat.Format16bppGrayScale, srcptr);

            ccDefects(srcbmp);

            return fingerNum;

        }

However I am getting exception

as follows

ArgumentException: A null reference or invalid value was found [GDI+ status: InvalidParameter]

System.Drawing.GDIPlus.CheckStatus (Status status)

System.Drawing.Bitmap..ctor (Int32 width, Int32 height, Int32 stride, PixelFormat format, IntPtr scan0)

(wrapper remoting-invoke-with-check) System.Drawing.Bitmap:.ctor (int,int,int,System.Drawing.Imaging.PixelFormat,intptr)

handDetectionDLL.handDetection.detectHandPose (IntPtr srcptr, Int32 w, Int32 h)

OpenNIDepthmapViewer.FixedUpdate () (at Assets/OpenNI/Scripts/OpenNIDepthmapViewer.cs:152)

Please help me get out of this problelm.

Thanks,

Kumar

[13Strider]

Did you end up getting this to work?

[Kumar Ahir]

Hi 13Strider,

Sorry for the late reply,

but yes I did make this work.

Thanks to all the folks for help.

[saman sannandeji]

That's great,  what method did you use to detect open/close  hand ?

[Kumar Ahir]

Hi,

Basically I followed below steps

1. extract the image of hand from imagemap using depthmap. using depthmap check if hand is in within range of kinect, then for all pixels that are in range get pixels from imagemap.

2. send this image to c# dll via a function call (basically interop service)

3. in c# get hand contour

4. calculate convex hull and convexity defects

5. did some processing with number of convexity defects and convex hulls i.e if the number of defects are more than 2/3 than palm is open with fingers separate, else the palm is closed

1st step is for optimization, as I am sending only grey image with palm in white and background in black. this allows to process the hand pose image faster.

regards,

kumar