Best background subtraction technique with kinect

[Gabriele]

Hi to all.
I need to track an object moving in a static environment with the IR
kinect's camera.
The simplest idea should be to subtract the constant signal from the
depth stream using a sample image took before the tracking. As you
know, depth image is not stable because of the random emissions of the
IR light.
Could someone suggest me some solutions as a workaround?
I thought about "recording" some frames to get as accurate as
possibile but i don't know how much could be feasible.
Any ideas will be really appreciated.

Thank you

[Kyle McDonald]

it sounds like you're trying to do background subtraction with the IR
image only?

if that is the case, i would recommend putting some kind of diffuser
over the IR projector. a piece of paper, or some cellophane, works
well.

if you're using the depth image, i recommend doing a weighted average
over time using a feedback buffer. this just means creating a new
image that is (for example) 10% the current image and 90% all the
previous images. this is one way to smooth things out over time so you
don't have to worry about the stability of the depth image.

if you're using opencv, this kind of operation can be done with a
single function: cv::addWeighted.

kyle

[Gabriele]

Yes, i am about to use only the IR stream. Both your advice seem to be
good!
I will give a try soon.
So if i use the addweighted function i'll have to wait some time until
the image stabilize, then i could use this stable IR image as a
background for the later subtraction?

Thank you very much for the quick and good answer :)

[Kyle McDonald]

yes, you'll have to wait 'some time'. depending on your weights, that
just means more like 5 frames instead of 1 frame.

this technique will work smoothing for the IR image as well as the depth image.

but because it's a more general computer vision question, i recommend
asking for more info on another list. the opencv mailing list has a
lot of discussion surrounding these kinds of questions:

http://groups.yahoo.com/subscribe/OpenCV

best,
kyle

[Gabriele]

Of course, thanks again.

[Eric Chen]

This might be a stupid question to ask, but why not just set a
threshold
on each pixels of depth images? something like filter?
The only problem I see with this technique is that it might limit the
range
of which user can stand. Also, it will need extra computation before
tracking can be done.

[Nicolas Burrus]

For the depth image, you can always ignore changes involving pixels
without data (0 depth value). This way you can avoid the detection of
the flickering pixels in the depth image as changes.

[Kyle McDonald]

the advantage of using background subtraction over fixed depth
thresholding is that it allows for objects to be near the camera, or
for tracking in a complex scene.

imagine tracking against a wall tat is sloped relative to the camera.
let's say the left side is close to the camera, and the right side is
far. you can't use any single depth threshold to pick out a person
from this, you have to account for the fact that the background is
changing across the scene.

kyle

[Eric Chen]

Thank you for the great explanation!!

[Gabriele]

I'm not sure i'm getting what you're saying..If i have pixels that
change their value in the IR stream, how can i expect a fixed 0 depth
value for them?

Kyle explained very well why threshold is not a valid option for
example in my case

[Nicolas Burrus]

On thin structures or when you have shadow, the depth image contains 0
values on these areas. I was just commenting that you can ignore them
if you want to do static background estimation by analyzing changes on
the depth image.

[Vasilie]

Maybe a threshold algorithm isn't so unnusable for your problem.
The Otsu treshold algorrithm allowes multiple thresholding so you can
figure out sloped walls in your image and you don't have to wait at
start up.

[Gabriele]

Still wouldn't be a solution..I want to allow my object to move all
around the room, from walls to the closest point reachable by the
kinect.
I've done this way:

1) Acquisition of the background: i let the kinect take all the
signals it could get from the IR stream. Depending on the condition it
could take some seconds or some more time (minutes)...Everytime it
resolves a previously 0-depth pixel i added the new value to the
image. I save the closest value it can get from each pixel it
resolves.
2) Use this image as a background and subtract it from the later
acquisitions. In this way i end up with a depth image containing only
the new objects which have been added after the first step.
It works quiet well, if you allow a medianblur in step 2 to eliminate
some shotnoise from the not-so-perfect background image.
Or at least, it fits my need.

[Ian Medeiros]

Your alghorithm sounds very good.
Do you have an image of your results to share with us?

2011/5/4 Gabriele <gabriele.f...@gmail.com>

[Gabriele]

Sure, here we are:

http://postimage.org/image/2dfqzyi3o/

Top left is the background image i created with the algorithm i
explained earlier. Top right the depth streaming.
Bottom left the rgb streaming and bottom right the result.
As you can "almost" see, i added the chair after the background
creation, so in the result appears only the new object. it is smoothed
because of the medianblur.
The procedure is far from perfect. Firts, i use the opencv wrapper on
the wiki. It's good because is based on the latest release of opencv,
but it seems it could be improved (don't know how, though).
Then, sometimes kinect takes closer or further depth values than the
real ones (even outside the accuracy) and this variations are not
negligible. A sort of occurrence filter should be implemented for
really best results in that case.

On May 4, 11:36 pm, Ian Medeiros <ianmcoe...@gmail.com> wrote:
> Your alghorithm sounds very good.
> Do you have an image of your results to share with us?
>

> 2011/5/4 Gabriele <gabriele.fontanar...@gmail.com>

[Radu Bogdan Rusu]

This might help you:

http://www.pointclouds.org/documentation/tutorials/octree_change.php

Cheers,
Radu.
--
Point Cloud Library (PCL) - http://pointclouds.org

[Gabriele]

Sound interesting, if i had to start again i would try it for sure.
Now i am fine with my solution, again with the limits i stated above..

Thanks :)

On 11 Mag, 22:55, Radu Bogdan Rusu <r...@willowgarage.com> wrote:
> This might help you:
>
> http://www.pointclouds.org/documentation/tutorials/octree_change.php
>
> Cheers,
> Radu.
> --

> Point Cloud Library (PCL) -http://pointclouds.org