Info on Depth Sensor's FOV and projective transform...

[Ixstala]

I wanted to display the point cloud data from the Kinect's depth
sensor with the rgb colors overlayed - as I've seen on several youtube
videos. However, using the built in function
"ConvertProjectiveToRealWorld(1,Apoint,Apoint)" slows my system to a
crawl (<3fps) when displaying all the points with measurable depth.
Without the transform I get around 30FPS drawing points, although the
image is incredibly distorted.

After some digging in the API I found the field of view for the depth
camera:

FovH=1.0144686707507438 (rad)
FovV=0.78980943449644714 (rad)

From there you can find the real world coordinates from the depth
cloud like so:

X_rw=(X_proj/Xres-.5)*Z*XtoZ
Y_rw=(0.5-Y_proj/Yres)*Z*YtoZ,
Z_rw=Z

where XtoZ=tan(FovH/2)*2, and YtoZ=tan(FovV/2)*2, and Z is the value
given by the depth map at each pixel.

Most of that can be precomputed and stored. Calculating the transform
myself gets me 30FPS with the full point cloud. Woo!

A better way would be to store a display list of the point cloud on a
640x480 grid then do vertex displacement mapping on the GPU with the
depth map, this would also include the simple calculation for the
projection into real world coordinates. That should be super fast.
Maybe this info will help someone?

[satori]

You should enable the AltViewPointCapability on the depthgenerator,
and openNI will automatically remap the depth pixels the the
apropriate screen space for the rgb feed. There is no need to do it
manually to get the right color for depth points from the rgb cam.

[satori]

Also... Look at the ConvertProjectiveToRealWorld() and
ConvertRealWorldToProjective() methods of the depthgenerator. They can
do screen space<-> worldspace conversion for you.