freenect_map_rgb_to_depth helper function
Florian Echtler
forgot to mention that I have created a patch to map RGB data to the
depth image, the opposite of the FREENECT_DEPTH_REGISTERED mode.
I've implemented this as a standalone helper function, as this operation
requires synchronized RGB and depth images (DEPTH_REGISTERED doesn't).
Since libfreenect doesn't deal with synchronization, this has to be done
in the main program.
Pull request is at https://github.com/OpenKinect/libfreenect/pull/284 -
opinions?
Florian
--
SENT FROM MY PDP-11
Ken Mankoff
I'm interested in this function, but don't fully understand it.
On Thu, 12 Apr 2012, Florian Echtler wrote:
> forgot to mention that I have created a patch to map RGB data to
> the depth image, the opposite of the FREENECT_DEPTH_REGISTERED
> mode.
It seems that your function needs the depth in mm units, but
unregistered. So one would call "freenect_apply_depth_to_mm" on the
raw data, then pass it to your function, along with an RGB image.
After, the RGB is aligned with depth. But do we know world x and y
yet? It doesn't look like it. We have x,y,z,r,g,b all aligned, and z
is world (mm), but x and y are still in pixel units.
So, we can call freenect_camera_to_world. But is this still valid
when the depth data has not been registered? Normally I think the
registration is applied and then freenect_camera_to_world is called.
Cheers,
-k.
Florian Echtler
On 12.04.2012 15:50, Ken Mankoff wrote:
> It seems that your function needs the depth in mm units, but
> unregistered. So one would call "freenect_apply_depth_to_mm" on the raw
> data, then pass it to your function, along with an RGB image. After, the
> RGB is aligned with depth. But do we know world x and y yet? It doesn't
> look like it. We have x,y,z,r,g,b all aligned, and z is world (mm), but
> x and y are still in pixel units.
Yes, you'll have to use depth data from the FREENECT_DEPTH_MM mode. The
depth data will remain unmodified, and freenect_camera_to_world
references the original depth image as intended.
> So, we can call freenect_camera_to_world. But is this still valid when
> the depth data has not been registered? Normally I think the
> registration is applied and then freenect_camera_to_world is called.
Actually, it's the other way round: when registering the depth data to
the RGB image, it's slightly distorted in the process and
freenect_camera_to_world will output distorted results. So if you want
to have world-aligned depth and RGB data, FREENECT_DEPTH_REGISTERED
isn't the best solution - using this helper function should be more
accurate.
Ken Mankoff
On Thu, 12 Apr 2012, Florian Echtler wrote:
OK. Good to know. Thanks. I'll be incorporating this into my offline
registration code soon.
-k.
Ken Mankoff
> On 12.04.2012 15:50, Ken Mankoff wrote:
>> It seems that your function needs the depth in mm units, but
>> unregistered. So one would call "freenect_apply_depth_to_mm" on
>> the raw data, then pass it to your function, along with an RGB
>> image. After, the RGB is aligned with depth. But do we know world
>> x and y yet? It doesn't look like it. We have x,y,z,r,g,b all
>> aligned, and z is world (mm), but x and y are still in pixel
>> units.
>
> Yes, you'll have to use depth data from the FREENECT_DEPTH_MM
> mode. The depth data will remain unmodified, and
> freenect_camera_to_world references the original depth image as
> intended.
>
What if I don't want to use data from FREENECT_DEPTH_MM. I prefer to
use raw (FREENECT_DEPTH_11BIT). Can I load raw data, call
freenect_apply_depth_to_mm, then pass the output of that to your
function?
Also, what is the difference between "freenect_apply_depth_to_mm"
and "freenect_raw_to_mm"? The latter says it converts raw to mm.
The former says it is the same as apply_registration, but without
alignment. Isn't registering without aligning the same as just
converting raw to mm?
-k.
Florian Echtler
> On Thu, 12 Apr 2012, Florian Echtler wrote:
>> On 12.04.2012 15:50, Ken Mankoff wrote:
>>> It seems that your function needs the depth in mm units, but
>>> unregistered. So one would call "freenect_apply_depth_to_mm" on the
>>> raw data, then pass it to your function, along with an RGB image.
>>> After, the RGB is aligned with depth. But do we know world x and y
>>> yet? It doesn't look like it. We have x,y,z,r,g,b all aligned, and z
>>> is world (mm), but x and y are still in pixel units.
>> Yes, you'll have to use depth data from the FREENECT_DEPTH_MM mode.
>> The depth data will remain unmodified, and freenect_camera_to_world
>> references the original depth image as intended.
> What if I don't want to use data from FREENECT_DEPTH_MM. I prefer to use
> raw (FREENECT_DEPTH_11BIT). Can I load raw data, call
> freenect_apply_depth_to_mm, then pass the output of that to your function?
Yes, there shouldn't be any difference in the result. However, using
FREENECT_DEPTH_MM will save you one buffer copy per frame - so I'd
suggest using that if you don't explicitly need the raw buffer.
> Also, what is the difference between "freenect_apply_depth_to_mm" and
> "freenect_raw_to_mm"? The latter says it converts raw to mm. The former
> says it is the same as apply_registration, but without alignment. Isn't
> registering without aligning the same as just converting raw to mm?
Correct. freenect_raw_to_mm is just an internal helper function used to
generate the lookup table that is later used both by
freenect_apply_registration and freenect_apply_depth_to_mm.
Florian
--
SENT FROM MY DEC VT50 TERMINAL
