depth map - rgb image calibration

[k2]

Does anyone have any recent info on calibrating the depth image with
the RGB image from the camera ? I see a bunch of threads and some
spurious references on the topic but nothing that appears
straightforward. Is there any additional details or straightforward
method beyond what is outlined in:

http://openkinect.org/wiki/Imaging_Information

Seems like this is step one before you can really do anything else.

thanks

[Kyle McDonald]

this feature is now built into libfreenect unstable
https://github.com/OpenKinect/libfreenect/tree/unstable

here's the most relevant commit:

https://github.com/OpenKinect/libfreenect/commit/baef89f81be00d1e386a5f84b96a64530b47a50e

[Kevin Smith]

Thanks, Is there a description of method used for the implementation ?  - or does it just do the translate/scale method to get the images to align ?

[Kyle McDonald]

you'll have to look through the mailing list some more for a longer
description of the implementation.

in short, it builds two lookup tables.

one does translation/scaling/undistortion (image space modifications).
this only gets you so far.

the second uses the depth values to account for the disparity between
the color and ir cameras.

this technique uses numbers that the kinect reports, which originally
come from in-factory calibration and vary from kinect to kinect. our
interpretation of these numbers is based on reading the openni
registration implementation.

kyle

[Γιάννης Γράβεζας]

First you'll have to

#include <libfreenect-registration.h>

After you initialize the video modes you'll have to fetch a copy of the registration struct like this:

freenect_registration reg = freenect_copy_registration(f_dev);

Which you can use like that(index is between 0 and 640*480):

depth_in_mm = reg.raw_to_mm_shift[raw_depth_at_index]; // The raw_to_mm_shift table is generally useful when you need to convert depth values from raw->mm y = reg.registration_table[index][1];

x = (reg.registration_table[index][0] + reg.depth_to_rgb_shift[depth_in_mm]) >> 8;

--
bliss is ignorance

[Kevin Smith]

Thanks much for the description.  I will git a copy of the unstable version and try it out tonight...I am hoping there are some useful function calls that do something like:

int depth = freenect_depth_from_pixel( x, y );

and conversely, 

freenect_pixel_from_depth_map(x, y, &px, &py);

something like that...the first one is probably the more useful one though...

A related question...  the depth map is not as high of a resolution as the RGB map, correct ?  So one depth map entry may map to more than one pixel (would be 4 pixels if the depth map is half the width and height).  I am wondering if there is some interpolation algorithms that could be used that look at neighboring pixels to interpolate (sort of anti-alias, the depth values)  Just a thought.

[Γιάννης Γράβεζας]

On Wed, Nov 23, 2011 at 3:32 AM, Kevin Smith <k2ms...@gmail.com> wrote:

Thanks much for the description.  I will git a copy of the unstable version and try it out tonight...I am hoping there are some useful function calls that do something like:

int depth = freenect_depth_from_pixel( x, y );

I don't think there's a function like that but you can easily calculate the 1D index from x,y (640*y+x) and use that to get the depth value from the buffer provided in the callback. 

A related question...  the depth map is not as high of a resolution as the RGB map, correct ?  So one depth map entry may map to more than one pixel (would be 4 pixels if the depth map is half the width and height).  I am wondering if there is some interpolation algorithms that could be used that look at neighboring pixels to interpolate (sort of anti-alias, the depth values)  Just a thought.

Registration will work correctly only if you setup the RGB video mode to 640*480, matching the depth mode. 

The higher video mode(1280*1024) is not exactly 2x2 the low mode, it would have to be 1280*960 for that.

 

--
bliss is ignorance

[Kevin Smith]

I tried the regview.c test program and I see that it does align the RGB image with the depth map but it appears to be off by a few pixels.  (for example - look at the finger tips when hand is in front of camera).  Is that because it's assuming a standard calibration ?

[jeff kramer]

It is a standard calibration, based on the data on each Kinect.

[drew.m...@gmail.com]

On Wed, Nov 23, 2011 at 20:47, Kevin Smith <k2ms...@gmail.com> wrote:
> I tried the regview.c test program and I see that it does align the RGB
> image with the depth map but it appears to be off by a few pixels.  (for
> example - look at the finger tips when hand is in front of camera).  Is that
> because it's assuming a standard calibration ?

In part. Note that the source depth image from which we derive the
aligned depth image is not perfect - your fingertips may not appear
the same in the depth image as they do in the IR image. If the Kinect
can't figure out which IR dot in the dot pattern is on your
fingertips, or if no dots hit your fingertips, then as far as the
depth camera is concerned, you're invisible. This is also why some
surfaces like most matte computer screens appear to produce no depth
data - the IR camera fails to see a reflection of the dot pattern.

Since the Kinect can't reliably see everything, it makes conservative
guesses that tend to make things look like they're a few pixels
smaller around the edges than they actually are.

How far off are we talking, here? Given a screenshot, I can probably
tell you if it seems like normal variation or error out of the
ordinary.

-Drew