Kinect Face
Felicity
I went through the Kinect SDK and Toolkit provided by Microsoft. Tested the Face Detection Sample, it worked successfully. But, how to recognize the faces ? I know the basics of OpenCV (VS2010). Is there any Kinect Libraries for face recognition? if no, what are the possible solutions? Are there, any tutorials available for face recognition using Kinect?
While that worked OK, I thought I could do better since the Kinect has such awesome face tracking. I ended up using the Kinect to find the face boundaries, crop it, and pass it into that library for recognition. I've cleaned up the code and put it out on github, hopefully it'll help someone else:
I've found project which could be a good source for you - -recognize-you/ but unfortunetly(for you) its homepage is not in english. The most important parts:
-project(with source code) is at -recognize-you/downloads/list
-in bibliography author mentioned this site - This seems to be a good start point for you.
There are no built in functionality for the Kinect that will provide face recognition. I'm not aware of any tutorials out there that will do it, but someone I'm sure has tried. It is on my short list; hopefully time will allow soon.
I would try saving the face tracking information and doing a comparison with that for recognition. You would have a "setup" function that would ask the user the stare at the Kinect, and would save the points the face tracker returns to you. When you wish to recognize a face, the user would look at the screen and you would compare the face tracker points to a database of faces. This is roughly how the Xbox does it.
The big trick is confidence levels. Numbers will not come back exactly as they did previously, so you will need to include buffers of values for each feature -- the code would then come back with "I'm 93% sure this is Bob".
What if your computer could detect your eyes, nose, and mouth? What if an application could understand your facial expressions? What if you could build such applications with minimal effort? Until today, if you wanted to create an accurate mechanism for annotating real-time facial characteristics, you should play with OpenCV and spend a ton of time experimenting with various algorithms and advanced machine vision concepts.
Navigate to the .xaml.cs file and place the following code in the constructor, just below the InitializeComponent method. The _colorReader is used to display the RGB stream (refer to my previous article about the different streams). The body reader is used to acquire the body data. We need the body data, since each face corresponds to a specific body instance.
Initialization of the color and depth readers is straightforward. Initializing the face reader is a little more tricky though. You need to explicitly state which face features you expect. Unless you feed the reader with a number of features, it will give you nothing! For our example, I have specified 9 elements (bounding box in color space, glasses, closed eyes, mouth open, etc.). You can add or remove features from the enumeration FaceFrameFeatures.
A face object is related to a corresponding body object (obviously). So, the face source should be updated with the tracking ID of the body. The following code detects the default body (if any) and assigns its unique tracking identifier to the face frame source.
Now is the time for the best part. Since the face source is connected with the body, we can specify what happens when a face frame is available. Face frames work exactly like the color, depth, infrared, and body frames: firstly, you get a reference to the frame, then you acquire the frame, and, if the frame is not empty, you can grab the FaceFrameResult object. The FaceFrameResult object encapsulates all of the available facial information.
To make our project more engaging, you can hide an eye ellipse if the eye is closed. Moreover, you can increase the size of the mouth ellipse if the mouth is open, and decrease its height if the mouth is closed.
Recently, we decided to extend our Kinetisense project with advanced facial capabilities. More specifically, we needed to access more facial points, including lips, jaw and cheeks. Moreover, we needed the X, Y and Z position of each point in the 3D space. Kinect Face API could not help us, since it was very limited for our scope of work.
At the time of writing, HD Face is the most advanced face tracking library out there. Not only does it detect the human face, but it also allows you to access over 1,000 facial points in the 3D space. Real-time. Within a few milliseconds. Not convinced? I developed a basic program that displays all of these points. Creepy, huh?!
The user interface is pretty simple. Open your MainWindow.xaml or MainPage.xaml file and place a drawing canvas within your grid. Preferably, you should add the canvas within a Viewbox element. The Viewbox element will let your Canvas scale proportionally as the window size changes. No additional effort from your side.
The next step is a little tricky. This is how we connect a body to a face. How do we do this? Simply by setting the TrackingId property of the Face source. The TrackingId is the same to TrackingId of the body.
Dive into the FaceReader_FrameArrived event handler. We need to check for two conditions before accessing any data. First we, need to ensure that the frame is not null. Secondly, we ensure that the frame has at least one tracked face. Ensuring these conditions, we can call the GetAndRefreshFaceAlignmentResult method, which updates the facial points and properties.
OK, we drew the points on screen. So what? Is there a way to actually understand what each point is? How can we identify where they eyes are? How can we detect the jaw? The API has no built-in mechanism to get a human-friendly representation of all of the face data. We need to handle over 1,000 points in the 3D space manually!
If you wish, you can use the Color, Depth, or Infrared bitmap generator and display the camera view behind the face. Keep in mind that simultaneous bitmap and face rendering may cause performance issues in your application. So, handle with care and do not over-use your resources.
Vangos Pterneas is a software engineer, book author, and award-winning Microsoft Most Valuable Professional (2014-2019). Since 2012, Vangos has been helping Fortune-500 companies and ambitious startups create demanding motion-tracking applications. He's obsessed with analyzing and modeling every aspect of human motion using AI and Maths. Vangos shares his passion by regularly publishing articles and open-source projects to help and inspire fellow developers.
Hi Georgios. Thank you very much for your kind words. HD Face is accurate when you are facing the sensor (frontal view). It remains accurate for approximately 45-60 degrees of head rotation. Expect a lot of false positives when you rotate your head for more than 60 degrees.
var forehead = vertices[HighDetailFacePoints.ForeheadCenter];
var cheekLeft = vertices[HighDetailFacePoints.LeftcheekCenter];
var cheekRight = vertices[HighDetailFacePoints.RightcheekCenter];
var jaw = vertices[HighDetailFacePoints.LowerjawLeftend];
You now have the 3D coordinates of 4 points. Each point is a CameraSpacePoint in the world space. Using Coordinate Mapper, you can map the world space to the RGB or depth/infrared space and find the 2D coordinates in a 19201080 or 512424 frame.
I have an interesting problem in which I am trying to detect if a person has a beard and a particular kind of glasses. Firstly, my plan with the beard is to use the HD face data in combination with the colour data and run the result through some sort of external imaging package (likely Emgu CV which is a wrapper for OpenCV). For the glasses, a similar trick will be used. The FaceFrameFeatures enumeration seems to only yield the presence of glasses and not whether they are the correct ones.
Do you know if it is possible to modify the code that is used to identify a face? I am trying to track infants (6-9 months) and Kinect does not pick up their faces. I suspect that this is because their features have different proportions than adults. I would like to try to modify the code to identify infant faces. Thanks!
Hi E B. We cannot modify the face tracking code. It is part of the SDK. Have you verified that the body of an infant is tracked? if the infant is lying on a bed, Kinect will not easily recognize it. You can put the sensor on the ceiling for better results.
Thank you for sharing ,sir! Here I want to ask you a problem.
If I want to put a obj glass model draw by opengl on the kinect face,how can I put the right location? That is, how can the obj model tracking the facepoint?
Hi Evan. Do not compare vertices. Instead, compare the relative distances between specific points. For example, you can compare the distance between the eyes, the jaw-forehead, etc. Also, remember to take the measurements at a specific angle. It will be much more accurate if you compare measurements between 10 degrees of rotation than 40 degrees of rotation.
Thank you for responding so quickly. I guess I should clarify though: I am looking at the relative distances between vertices by calculating the 3D vector length (SQRT((x2-x1)^2 + (y2-y1)^2 +(z2-z1)^2) between two points (say tip of the nose to the left eye) and then comparing that length to the length of the same vector for a different face. The results, as I mentioned, were not accurate. Are you saying there are other values I should be getting?
Hi Vangos,
I already tried this programme as per your recommendation. As I already assembled the code without error, unfortunately the program ran but nothing coming on the UI canvas. Just Kinect 2 Face HD text. Can you help me on this one?
Thank you.
Thank you for your response. Averaging frames might solve face flickering. But im still now sure how to use these new smoother positions to acquire smoother AnimationUnits (FaceAlignment.AnimationUnits).
4a15465005