Finger Tracking

[ThreeWillow]

I am trying to  track fingers in a video, and using the following methods, as attached:

 

The first step is to segment hands from the still image, then use features of fingertips to find fingertips.

One of the feature of the finger is that there is a circle with diameter of  the width of the finger at the front of every finger,

so I need to check for every candidate fingertip point that it is surrounded by a circle.

 

Can any tell me how to do it?

 

[Iker Saint]

Probably you should use OpenCV for image analisis, a good start is to take the initial frame, you have to check for the contours of the images, then the hulls, verify the epsilon of every hull; finally on the post frames you have to use the function goodfeaturestotrack  and verify for coincidences with your previous detected fingers; in this couple of videos I implement different approaches to this techniques http://www.youtube.com/watch?v=PW0peJNvwsU, http://www.youtube.com/watch?v=-qCzzt40gZY.

Best Regards,

Iker

[Ignacio Avellino]

Hey Iker!

It would be great if you could share some code.

All forums I roam regarding this mater provide the solution as you've mentioned, but it's hard to get hold of a code snippet.

Since I've been looking at this problem since some time, herex is my input. I'll post some code below, which is pieces of the code I'm using, so forgive me if it doesn't compile out of the box. But you get an idea of what to do.

So far what I'm doing is the following (to a 2d image):

vector<vector<cv::Point> > contours;

vector<Vec4i> hierarchy;

Mat thresholdOutput;

// Threshold first

threshold( *fingerMatImage, thresholdOutput, thresh, 255, THRESH_BINARY );

// Find contours

findContours( thresholdOutput, contours, hierarchy, CV_RETR_TREE, CV_CHAIN_APPROX_SIMPLE, cv::Point(0, 0) );

vector<vector<cv::Point> > hullsP(contours.size());

double maxArea = 0;

    int contourMaxAreaId = -1;

    // Find biggest contour

    for(int i = 0; i <contours.size(); i++){

        // Find the hulls

        convexHull(contours[i], hullsP[i], false, true);

        

        // Find the defects

        if (contours[i].size() >3 )

        {

            // Area of the hull

            //double area = cv::contourArea(hullsP[i]);

            double area = cv::contourArea(contours[i]);

            if (area > 1000 && area < 60000 && area > maxArea)  {

                maxArea = area;

                contourMaxAreaId = i;

            }

        }

    }

    

    for( int i = 0; i< contours.size(); i++ )

    {

        drawContours(*touchesHullsImage, hullsP, i, CV_RGB(255, 255, 255), 1, 8, vector<Vec4i>(), 0, cv::Point());

    }

For some theory, you might want to read: http://tmroyal.com/a-high-level-description-of-two-fingertip-tracking-techniques-k-curvature-and-convexity-defects/

Cheers

[Iker Saint]

There is a little piece of code that could be of help for you:

http://pastebin.com/GGL10UHp

On the previous code you can see a lot of things, first; I have a function "GetHandSquare" that function calculates the square on what the hand is contained, is pretty easy beacuse you have the realworld coordinates, that coordinates are in milimeters, the avergae hand is 189 mm long, so you use just 250 mm to get a saqure around the Nite point that you have; then you have to create a virtual matrix that is just for the hand (Speed thing), that cv::Mat will be your hand only image, his type is CV_16UC1, and then you can start to find fingers, the code below helps you with that:

http://pastebin.com/CQdq9J4f

That code is not mine, so i have no testes, just refactor the code and complete a couple of lines that I think improve the functionality, be careful specially with "cv::approxPolyDP( ContourMat, ApproxCurve, 20, true);" the distance of user to the sensor has to be extrapolated to the parameter epsilon (20), because you probably have a little image that doesn't have a epsilon of 20 in any finger, so do you have to change the value in order to work with he distance; if you work with very small images you have to checkout the hulls, beause probably you only find two or three and there is no possibility to build a correct angle ("acos( (v1.x * v2.x + v1.y * v2.y) / (norm( v1 ) * norm( v2 )) )" is not going to work);using part of your code you have to implement a little correction algorithm if you use threshold or gaussian blur, etc..; that is because the point that you find like a finger could be have a wrong distance value, the gaussian blur for example adds 3 pixels of deviation if you use 5 like parameter; i strongly recommends to you, use the direction vector and distance to find quickly the previous finger in the next image, that could speed a little bit the detection.

Finally, I use on my own projects of finger detection (without Nite) OpenCV, first processing the image, extracting the contour and interesting points, using gaussian blur, processing the color image (not the deep) and then using goodfeaturestotrack, that allows me to track the finger without the problems of the distance and the vector direction; sadly I'm not allowed to share the code.

Some interesting documentation:

http://www.aishack.in/topics/tutorials/vision/page/6/

http://www.aishack.in/2010/05/corner-detection-in-opencv/

http://es.scribd.com/doc/46059251/OpenCV-Lections-5-Smoothing

Best Regards,