Cyril Thibout
I’m
interested in replacing the keyboard in VR games (using an occulus headset for instance) with full body gesture
tracking.
I droped the treadmills options since it would require omnidirectonial
treadmills to work with headsets since the user can turns in any orientation.
The kinect may be a solution: leaning forwards or backwards is intuitive enough
for walking/running. leaning left and right would be stafing. Crouching is easy
with the kinect too. Combined with the headset tracker, we would have all the 6
DOF required.
the main issue for me now is: can a kinect still track the body motion when the
body is NOT facing the kinect?
since the user would wear a headset he rapidely can be in any orientation from
the kinect. the kinect would map body gestures/positions to keyboard buttons
Can a Kinect track the motion of my arm when my body is 90° oriented from the
Kinect? In this case one arm is partly hidden by the body.
What if I ‘m at 180° : will it understand all the motions are inverted ?
May be there is some piece of software on the net that already do the job?
Jan Ciger
the main issue for me now is: can a kinect still track the body motion when the body is NOT facing the kinect?
Yes, it can - once it acquires "lock" on the user (fits the skeleton), it will continue tracking. Of course, if you are sideways towards to the camera, the accuracy will not be great and it cannot extrapolate things it doesn't see (e.g. the pose of your arm that is away from the camera).
since the user would wear a headset he rapidely can be in any orientation from the kinect. the kinect would map body gestures/positions to keyboard buttons
Can a Kinect track the motion of my arm when my body is 90° oriented from the Kinect? In this case one arm is partly hidden by the body.
What if I ‘m at 180° : will it understand all the motions are inverted ?
It will work and not, at 180 it won't be "inverted" unless you have entered the field of view of the Kinect with your back to the camera (and the tracking got initialized backwards) - it can track a full circle rotation, of course not very accurately. However, as I said above - don't expect great performance, single camera can do only so much. You cannot track something you don't see, only extrapolate - that is ok for a short occlusion but it will produce big errors fast unless the occlusion is resolved.
May be there is some piece of software on the net that already do the job?
FAAST, OpenNI, the MS SDK for Kinect all allow tracking with Kinect. However, if you want to track reliably in the presence of occlusions (even your hand occluded by your own body from the camera), you need redundant cameras. That is why commercial optical trackers have always more than 2 cameras (which are the minimum for 3D tracking) - the rest is (mainly) for redundancy, if one camera doesn't see your hand, another one will and the system can continue tracking it.
Unfortunately getting two Kinects to work well in the same room is highly non-trivial, because they will interfere with each other (get "confused" which projected dot belongs to which Kinect). I suppose it can be compensated for somehow, but it is a very complex problem. The only software that can do decent tracking with two Kinects is iPi (http://www.ipisoft.com/), but that is not realtime - there is a heavy calculation/processing stage that is done offline on the recorded video to extract the motion. iPi doesn't use the OpenNI/MS SDK skeletons and uses its own image processing and skeleton fitting algorithms - you basically record the video from each Kinect + depth map using one application and then use another one for processing the files offline. Also, your setup must be calibrated (it needs to know the mutual position of the Kinects) - there is a procedure for it.
Regards,
Jan
Naem BARON
Jan Ciger
Hello world,
I am planning to achieve the same thing using a kinect and an IMU.The kinect can track the body from backside but I am not sure if it can discern the side (thus invert left/right).If I remember well, it has a lot of difficulties when looking at sideways.
From the experiments we did, Kinect was able to keep tracking when I was turning left and right, but with a lot of noise and errors in the data. It is OK for something like looking to the left or right and then turning back, but not if you stay turned with your side to your camera - the tracking will go unusable fast, because it doesn't seen enough of your body to fit the skeleton reliably.
Jan