multiple kinects working

870 views
Skip to first unread message

Aurel W.

unread,
Nov 19, 2010, 5:40:22 PM11/19/10
to openk...@googlegroups.com
Hello,

in the last days there has been quite some discussion about using
multiple kinects. Main reason is, we could get rid of occlusions and
really use it for full real time 3d reconstruction. A first starting
point would be to use an stereo setup.

Sorry for this long post, but I kept working on this like crazy in the
past two days so I have quite some stuff to say, hope it turns out
useful to you. Interesting stuff on the bottom ;)

When you think of, how the depth sensor works one issue comes
immediately to your mind, A structured light pattern is projecteder in
IR, while the viewing frustum of the projector has some slight offset
to the one of an ir camera, but they are axis aligned. From just this
it is possible to reconstruct depth. However, if you would use
multiple kinects, there could be interferences, when the structured
light is visible for the other kinect or when the laser projector is
directly visible without any reflection. So it was common sens, that
multiple kinects won't work by now.

Some ideas to address this issue were stated, not all of them are
practical. Some comments on those ideas:

* Time multiplexing with 1/30s time slots. This basically means that
in a stereo setup, you would subsequently capture one frame at one
kinect and disable the laser projector on the other kinect. First keep
in mind that all devices are running asynchronously. Depending on the
exposure time for the ir-camera, one frame from one kinect can
"overlap" two frames in the other kinect. In the worst cast of 1/30s
exposure time, we would need to wait for an entire frame, before
starting to capture. This means, in a stereo setup we would drop from
the orifinal 30fps, to 7.5fps. lame. In addition, we don't know yet,
if the device allows us to enable and disable the ir projector frame
by frame. If not, someone would need to do this by altering hardware
and switching the diode separately with a micro controler (or the
built in motor controler, if you want to be clever :)

* Frequency multiplexing. Using different narrow band ir lasers and
filters. This would definitely work, if you get the optics right. Tho
quite complicated imho.

* Polarization. Won't work. A reflection on an arbitrary surface won't
preserve polarization well enough. The reason why polarization
projection screens for e.g. work, is because they use a special
coating. This won't work on human skin or other materials tho.

Because all this ideas have some sort of issue I tried something
different. Normally someone would use synchronized devices, very short
exposure times and time multiplexing. But that won't be possible with
the kinect. My approach now was to add separate shutters to both, the
camera and projector and to do time multiplexing within one single
frame of the ir camera. This can be done with a high speed shutter.

For the ir-projector the best approach would be to modulate the laser
diode accordingly. For the camera, some sort of mechanical or lcd
shutter is needed. Shutters would be added to both kinects and the
shutters have to be synchronized of course, while the ir-cameras of
the kinects are running asynchronously. Of course, the exposure time
of one single frame is shorter now.

To find out, how the depth sensor would behave, if i add a shutter, i
made different experiments.This is only with one kinect by now. One
setup looked like this:

http://atommuell.mum.jku.at/~aurel/kinect_roling_shuter_proto_2.jpg

The depth sensor gives slightly more errors here, because of the short
exposure time, but in general the sensor seems to be pretty robust
when doing such things as adding rolling shutters.

The next step was to get a second kinect and to build an actual
prototype, where both sensors are stereo separated by shutters.
However, I was quite surprised when i pointed two plain kinects on the
same object and started capturing. My intention was, and as it was
discussed all around the net by now, that things would be completely
screwed up. Both sensors were working quite good with some added
errors. See for yourself:

http://atommuell.mum.jku.at/~aurel/two_kinects_same_direction_c_both.png

Compared to using just one kinect:

http://atommuell.mum.jku.at/~aurel/two_kinects_same_direction_c_single.png

Interference is imho very low, much lower than expected. Also an
interesting test, to lower the luminance of one of the projects to
50%:

http://atommuell.mum.jku.at/~aurel/two_kinects_same_direction_c_both_one_proj_shutter.png

I don't think that time multiplexing is completely of the table now,
but multiple kinects are working way better than expected, so maybe
the hardware could be used without any modification in some cases. We
also would have to see about interference when using more than two
kinects.

that's it for now,

aurel

Nink

unread,
Nov 19, 2010, 5:54:38 PM11/19/10
to openk...@googlegroups.com
What I find real intersting put two kinects on top of each other and cover the infared for one but leave camera uncovered. Black screen. They know what signal they sent out. There is no issue the folks at primesence were smart.

I have a quad core i7 running now and its humming nicely with 2 kinects. Trying to add third I have some pci cards I am playing with at the moment to get the third one up. Goal is 6. Peaks at 50 percent then idles at 20 percent. Ill tweet a pic on @nink.
Sent from my BlackBerry

Aurel W.

unread,
Nov 19, 2010, 6:01:54 PM11/19/10
to openk...@googlegroups.com
Another big issue, were interference is quite heavy, is when you face
two kinects in a way, that one ir camera has another laser projector
in the field of view. This results in some lens flare. So I guess
adding shutters and projector modulation may still be necessary for
good results in multi kinect setups.

Radu Bogdan Rusu

unread,
Nov 19, 2010, 7:15:56 PM11/19/10
to openk...@googlegroups.com, Aurel W.
Again, in our tests the interference is minimal (if any) with 2 kinects. We'll post a video up later to show this in
case people still have doubts ;) We got about 10-20 of them by now in the building, so we might try a crazy 20-Kinect
registration experiment ;)

Cheers,
Radu.

Nink

unread,
Nov 19, 2010, 8:41:47 PM11/19/10
to openk...@googlegroups.com
Radu. I can't seem to get the third kinect to start. I am using the latest ROS build and on seperate bus for each one. I have tried plugging in one at a time, different orders etc but 3rd kinect still crashes each time. It sees the third kinect tries to initialize and that's about it.

Any advice appreciated

Radu Bogdan Rusu

unread,
Nov 20, 2010, 12:14:23 AM11/20/10
to Nink, ros-kinect, openk...@googlegroups.com
Nink,

Interesting - ok I have to admit that I haven't tried 3 kinects yet on my system. I think that each of them might
require a separate USB controller, as we had issues with 2 on the same controller earlier today. Can you make sure
they're plugged in different controllers?

Cheers,
Radu.

Nink

unread,
Nov 20, 2010, 10:38:05 AM11/20/10
to Radu Bogdan Rusu, ros-kinect, openk...@googlegroups.com
I purchased 2 pci usb cards and Libusb tells me all on seperate bus. Joshb had a great idea send initiation code to each card before we start collecting data to make sure we can at least activate the ir projector.

If someone can do me a big favour and just create a cut down version of ROS libfreenect that just sends the inital request to activate the kinect would be great.

Thanks

Nink

unread,
Nov 21, 2010, 4:43:40 PM11/21/10
to OpenKinect
Hi Radu

I ahve not made any progress on the multiple kinects. Although it
identifies 3 kinects it doesnt address each one individually so I
think we are trying to initialise the same one over again. I have
tried to just init all 3 but no go. Perhaps if we had a way to select
what device we want to address as a switch on glview (say K1 K2 K3 or
K4 etc) to ensure it knows what device we want to communicate with.

Here is my config as you can see they are all on a seperate bus and
all high speed

Bus 03.Port 1: Dev 1, Class=root_hub, Driver=ehci_hcd/4p, 480M
|__ Port 1: Dev 6, If 0, Class=hub, Driver=hub/3p, 480M
|__ Port 1: Dev 8, If 0, Class=vend., Driver=, 480M
|__ Port 2: Dev 7, If 0, Class=vend., Driver=, 12M
|__ Port 3: Dev 9, If 0, Class=vend., Driver=, 480M
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci_hcd/6p, 480M
|__ Port 6: Dev 3, If 0, Class=hub, Driver=hub/3p, 480M
|__ Port 1: Dev 5, If 0, Class=vend., Driver=, 480M
|__ Port 2: Dev 4, If 0, Class=vend., Driver=, 12M
|__ Port 3: Dev 6, If 0, Class=vend., Driver=, 480M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci_hcd/6p, 480M
|__ Port 5: Dev 3, If 0, Class=hub, Driver=hub/3p, 480M
|__ Port 1: Dev 5, If 0, Class=vend., Driver=, 480M
|__ Port 2: Dev 4, If 0, Class=vend., Driver=, 12M
|__ Port 3: Dev 6, If 0, Class=vend., Driver=, 480M

On Nov 19, 7:15 pm, Radu Bogdan Rusu <r...@willowgarage.com> wrote:
> Again, in our tests the interference is minimal (if any) with 2kinects. We'll post a video up later to show this in
> case people still have doubts ;) We got about 10-20 of them by now in the building, so we might try a crazy 20-Kinect
> registration experiment ;)
>
> Cheers,
> Radu.
>
> On 11/19/2010 03:01 PM, Aurel W. wrote:
>
> > Another big issue, were interference is quite heavy, is when you face
> > twokinectsin a way, that one ir camera has another laser projector
> > in the field of view. This results in some lens flare. So I guess
> > adding shutters and projector modulation may still be necessary for
> > good results in multi kinect setups.
>
> > On 19 November 2010 23:54, Nink<nink...@gmail.com>  wrote:
> >> What I find real intersting put twokinectson top of each other and cover the infared for one but leave camera uncovered.  Black screen.   They know what signal they sent out.  There is no issue the folks at primesence were smart.
>
> >> I have a quad core i7 running now and its humming nicely with 2kinects.  Trying to add third I have some pci cards I am playing with at the moment to get the third one up.  Goal is 6.  Peaks at 50 percent then idles at 20 percent.  Ill tweet a pic on @nink.
> >> Sent from my BlackBerry
>
> >> -----Original Message-----
> >> From: "Aurel W."<aure...@gmail.com>
> >> Sender: openk...@googlegroups.com
> >> Date: Fri, 19 Nov 2010 23:40:22
> >> To:<openk...@googlegroups.com>
> >> Reply-To: openk...@googlegroups.com
> >> Subject:multiplekinectsworking
>
> >> Hello,
>
> >> in the last days there has been quite some discussion about using
> >>multiplekinects. Main reason is, we could get rid of occlusions and
> >> really use it for full real time 3d reconstruction. A first starting
> >> point would be to use an stereo setup.
>
> >> Sorry for this long post, but I kept working on this like crazy in the
> >> past two days so I have quite some stuff to say, hope it turns out
> >> useful to you. Interesting stuff on the bottom ;)
>
> >> When you think of, how the depth sensor works one issue comes
> >> immediately to your mind, A structured light pattern is projecteder in
> >> IR, while the viewing frustum of the projector has some slight offset
> >> to the one of an ir camera, but they are axis aligned. From just this
> >> it is possible to reconstruct depth. However, if you would use
> >>multiplekinects, there could be interferences, when the structured
> >> light is visible for the other kinect or when the laser projector is
> >> directly visible without any reflection. So it was common sens, that
> >>multiplekinectswon't work by now.
> >> shutter is needed. Shutters would be added to bothkinectsand the
> >> shutters have to be synchronized of course, while the ir-cameras of
> >> thekinectsare running asynchronously. Of course, the exposure time
> >> of one single frame is shorter now.
>
> >> To find out, how the depth sensor would behave, if i add a shutter, i
> >> made different experiments.This is only with one kinect by now. One
> >> setup looked like this:
>
> >>http://atommuell.mum.jku.at/~aurel/kinect_roling_shuter_proto_2.jpg
>
> >> The depth sensor gives slightly more errors here, because of the short
> >> exposure time, but in general the sensor seems to be pretty robust
> >> when doing such things as adding rolling shutters.
>
> >> The next step was to get a second kinect and to build an actual
> >> prototype, where both sensors are stereo separated by shutters.
> >> However, I was quite surprised when i pointed two plainkinectson the
> >> same object and started capturing. My intention was, and as it was
> >> discussed all around the net by now, that things would be completely
> >> screwed up. Both sensors were working quite good with some added
> >> errors. See for yourself:
>
> >>http://atommuell.mum.jku.at/~aurel/two_kinects_same_direction_c_both.png
>
> >> Compared to using just one kinect:
>
> >>http://atommuell.mum.jku.at/~aurel/two_kinects_same_direction_c_singl...
>
> >> Interference is imho very low, much lower than expected. Also an
> >> interesting test, to lower the luminance of one of the projects to
> >> 50%:
>
> >>http://atommuell.mum.jku.at/~aurel/two_kinects_same_direction_c_both_...
>
> >> I don't think that time multiplexing is completely of the table now,
> >> butmultiplekinectsare working way better than expected, so maybe

Radu Bogdan Rusu

unread,
Nov 21, 2010, 4:48:45 PM11/21/10
to Nink, ros-kinect, openk...@googlegroups.com
Nink,

I only have 2 Kinects here, but I'll try with 3 or more on Monday at work.

Are you using ROS-kinect btw? We've already tested this and it works great there. You can easily do device selection as
a command line parameter or a ROS launch parameter, then pop up individual views of each camera in
rviz/image_view/pcl_visualization, etc. I am pretty sure that if you use that, you can get it to work for 3 cameras too,
unless there's some hardware limitation that we're not aware of.

Also make sure that whatever libfreenect fork you're using includes the multiple cameras patch that we wrote.

Cheers,
Radu.

BlueSol

unread,
Nov 23, 2010, 11:37:29 AM11/23/10
to OpenKinect
Any Update on multiple Kinects ( i.e. 3 + running)

Watching the MS Lightsense demo looks like the can do it with the
PrimeSense adapters.

Lightspace
http://research.microsoft.com/apps/video/default.aspx?id=139120

3 x PrimeSense Depth Sensing Cameras


On Nov 21, 4:48 pm, Radu Bogdan Rusu <r...@willowgarage.com> wrote:
> Nink,
>
> I only have 2Kinectshere, but I'll try with3or more on Monday at work.
>
> Are you using ROS-kinect btw? We've already tested this and it works great there. You can easily do device selection as
> a command line parameter or a ROS launch parameter, then pop up individual views of each camera in
> rviz/image_view/pcl_visualization, etc. I am pretty sure that if you use that, you can get it to work for3cameras too,
> unless there's some hardware limitation that we're not aware of.
>
> Also make sure that whatever libfreenect fork you're using includes the multiple cameras patch that we wrote.
>
> Cheers,
> Radu.
>
> On 11/21/2010 01:43 PM, Nink wrote:
>
> > Hi Radu
>
> > I ahve not made any progress on the multiplekinects.  Although it
> > identifies3kinectsit doesnt address each one individually so I
> > think we are trying to initialise the same one over again.  I have
> > tried to just init all3but no go.  Perhaps if we had a way to select
> > what device we want to address as a switch on glview (say K1 K2 K3 or
> > K4 etc) to ensure it knows what device we want to communicate with.
>
> > Here is my config as you can see they are all on a seperate bus and
> > all high speed
>
> >    Bus 03.Port 1: Dev 1, Class=root_hub, Driver=ehci_hcd/4p, 480M
> >      |__ Port 1: Dev 6, If 0, Class=hub, Driver=hub/3p, 480M
> >          |__ Port 1: Dev 8, If 0, Class=vend., Driver=, 480M
> >          |__ Port 2: Dev 7, If 0, Class=vend., Driver=, 12M
> >          |__ Port3: Dev 9, If 0, Class=vend., Driver=, 480M
> > /:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci_hcd/6p, 480M
> >      |__ Port 6: Dev3, If 0, Class=hub, Driver=hub/3p, 480M
> >          |__ Port 1: Dev 5, If 0, Class=vend., Driver=, 480M
> >          |__ Port 2: Dev 4, If 0, Class=vend., Driver=, 12M
> >          |__ Port3: Dev 6, If 0, Class=vend., Driver=, 480M
> > /:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci_hcd/6p, 480M
> >      |__ Port 5: Dev3, If 0, Class=hub, Driver=hub/3p, 480M
> >          |__ Port 1: Dev 5, If 0, Class=vend., Driver=, 480M
> >          |__ Port 2: Dev 4, If 0, Class=vend., Driver=, 12M
> >          |__ Port3: Dev 6, If 0, Class=vend., Driver=, 480M

Nink

unread,
Nov 23, 2010, 11:52:57 AM11/23/10
to openk...@googlegroups.com
I had 4 running 2 on 1pc and 1 on mac 1 on windows laptop in a circle with little interance. I still have not got 3 or 4 on 1 pc. I could not get ROS-kinect working although ROS glview did see 3 kinects it keeps trying to initialise the same ones when you run multiple sessions.

I am not allowed to look at source code (day job conflict) so I was hoping someone could update the python code to allow you to select kinect you want to communicate with.
Sent from my BlackBerry

-----Original Message-----
From: BlueSol <josh.j...@gmail.com>
Sender: openk...@googlegroups.com
Date: Tue, 23 Nov 2010 08:37:29
To: OpenKinect<openk...@googlegroups.com>
Reply-To: openk...@googlegroups.com
Subject: Re: multiple kinects working

Radu Bogdan Rusu

unread,
Nov 23, 2010, 12:25:14 PM11/23/10
to openk...@googlegroups.com, Nink
You have to make sure they are on separate USB controllers. That's the only way we got them to work so far.

Cheers,
Radu.

Nink

unread,
Nov 30, 2010, 3:20:03 PM11/30/10
to Radu Bogdan Rusu, openk...@googlegroups.com
Hey Radu I installed latest code and if you type ./glview 0 then ./glview 1 ./glview 2 etc all the kinects fire up nicely now on a single computer. I have only been able to get 4 up so far running at about 50% utilization on i7. I ran out of PCI slots for seperate USB hubs so I will try to order a multi bus USB card. Looks like 6 kinects is going to be fine.


Sent from my BlackBerry

-----Original Message-----
From: Radu Bogdan Rusu <ru...@willowgarage.com>
Date: Tue, 23 Nov 2010 09:25:14
To: <openk...@googlegroups.com>
Reply-To: Radu Bogdan Rusu <ru...@willowgarage.com>
Reply all
Reply to author
Forward
0 new messages