Xtion at UNIT_100UM

[MagWeb]

Seems you can use sub millimeter units also for the Xtion (here it appears as "Carmine 1.08" in the device manager). Maybe this works for the Kinect too if the OpenNi2 backend is used (not tested yet).

This does not make much sense if you use this "big size optimized" sensor without having added lenses but with lenses there is a noteable difference in its Z resolution.

I attached an image comparing single depth frame results.

[Christoph Heindl]

You are descovering everything, aren't you MagWeb? :) Actually the only sensor supporting sub mm resolution parameters is the Carmine 1.09. You can use that in reconstruction too, just that all the units are then expressed in 100um. 

[Tim Yoshi]

Can somebody explain what I should change in config files to make it work with 100UM unit?

I changed that in "Primesense Carmine 1.09 640x480.txt" but out of the box it seems to not working like that. Tried to change in "Head high detail.txt" minimum_corners and maximum corners by x10, but it gives me some 0.7 fps and nothing more. Is this normal because it should handle some additional data?

[Christoph Heindl]

Am Mittwoch, 3. April 2013 13:12:11 UTC+2 schrieb Tim Yoshi:

Can somebody explain what I should change in config files to make it work with 100UM unit?

I changed that in "Primesense Carmine 1.09 640x480.txt" but out of the box it seems to not working like that. Tried to change in "Head high detail.txt" minimum_corners and maximum corners by x10, but it gives me some 0.7 fps and nothing more. Is this normal because it should handle some additional data?

I don't have a config at hand right now since I'm out of office, but maybe MagWeb can provide one?

[MagWeb]

Yes, he can  

Seems Tim uses Carmine 1.09?

Here´s a set of config files (at a 0.5m volume) as used for my tests. It gives me about 6.5 fps here .

The sensor's config's intrinsics are at the default factory settings. 

To get a real plus from the UM settings you should calibrate and  replace these intrinsic parameters as explained here:

https://groups.google.com/forum/#!topic/reconstructme/PBXtn9Wax1Q

[Tim Yoshi]

thank you so much. I'll give this a try.

[MagWeb]

Oh, forgot:

Here ReMe is set to display the IR image as the sensor sees the projection. This is for my lenses often partially cover the RGB camera... You might undo this by replacing

aux_stream {

  type: STREAM_IR

by

aux_stream {

  type: STREAM_COLOR

in PrimeSenseCarmine 1.09 640x480_UM.txt

[Tim Yoshi]

Thanks, by my lenses also cover my Carmine 1.09. So it is better to stay with IR.

[Tim Yoshi]

My new GTX 660 Ti gives me on this setting some 17-23 fps. 

But I'm still having some mess with glasses. Tried to calibrate it with your recommendations  but this gave me even more distortion. Objects were looking very fat :) Tried to leave just measured raw calibration (as regular) - it gives more me less distortions, bit still there is a lot of noise and some "distant" (like nose or ears) objects tend to be extremely distorted. Although plain skin on the face gives pretty nice and detailed results.

Possibly I'm having to high expectations from it's possibilities. Or I'm a dumb (which is more likely) :)

On Wednesday, April 3, 2013 6:18:55 PM UTC+3, MagWeb wrote:

[Christoph Heindl]

Am Mittwoch, 3. April 2013 17:18:55 UTC+2 schrieb MagWeb:

Yes, he can  

Seems Tim uses Carmine 1.09?

Here´s a set of config files (at a 0.5m volume) as used for my tests. It gives me about 6.5 fps here .

Thanks for jumping in :) The frame-rate concerns me though. AFAIK you do have a quite powerful GPU.

[MagWeb]

@Tim

Seems the optical centers of the lenses are displaced?

What happens if you scan a flat plane (sensor looking almost perpendicular onto it) doing only a small movement with the sensor? Is this result somehow bowed?

If yes, I can not correct this with the current calibration tool. You have to try to find a better position for your lenses.

To give a rough orientation where the glasses' centers might be:

The average distance of human pupils is 65mm (male) or 60mm (female). It makes sense to use this standard distance to judge the optical centers of readymade glasses. The Primesense distance between sensor and projector is about 75mm. So you have to increase the lenses distance about 10mm compared to their distance in the glasses' frame.

Another thing is the right angle of the lenses. Due to the not symmetrical housing I have to fix the sensor lense's inner side about 3mm of  the IRSensor/RGB covering glass while its outer side is directly fixed at the housing...

Needs some experiments to find the right position....

[MagWeb]

Thanks for jumping in :) The frame-rate concerns me though. AFAIK you do have a quite powerful GPU.

Yes, Tim's frame rate made me worried too. I can live with the low frame rate but a bigger one would be nice....

I'm going to try other Catalyst driver versions but maybe this is due to the mobility version of my GPU...

[Tim Yoshi]

Ah, of course - previously I also tried to go with mobile GPU in my laptop it was kinda average guy - GT640M with core i5-3210 (2,5 GHz). But with regular XboX kinect with ordinary settings I was having like 5-6 fps. In SD mode it pumped up to 17 Fps.

Because of that I decided to go for desktop GPU and reanimate my old PC (2 Gb RAM, Core2Duo E8400 2x3,0 GHz) and bought desktop GTX 660 Ti. Now with Carmine 1.09 which able to deliver 60 FPS in 320x240 mode I tried it in this mode. So with "Volume 1 meter cubed high detail.txt" and "PrimeSense Carmine 1.09 320x240.txt manually set to "  fps: 60" i can achieve around 45-50 actual fps on my screen. If I select 640x480 mode, in HD it gives me around 20-25 fps. Now that is really nice and smooth experience. 

So comparing my laptop GPU and desktop GPU results in this test http://clbenchmark.com/compare.jsp?config_0=14868251&config_1=13051064 It is not so surprising - they have around x5-6 difference in pure OpenCL computational power. Also we may found that even humble Radeon 7850 which cost almost twice (1.67 actually in Ukraine) cheaper can give us around 50-100% boost in performance over my GTX 660 Ti. Also it extreme computational power can be very useful for some distributed scientific calculations or just to mine some bitcoins :)

So I will definitely change my nVidia to Radeon just when guys got it fixed. Hope it will be possible soon because I just bought it in monday, and by Ukrainian laws I can only change/return it only within 14 days of purchase.

So conclusions which I made thru all of this - only go for desktop Radeon GPUs. Also mobile ones above 7850M level could give almost the same performance as my desktop GTX660Ti and 7970M even boosts it to the level of above mentioned desktop 7850.

[Tim Yoshi]

Ah, now I see what I did wrong. I just put the whole glasses (except earpieces which was successfully sawed) in front of sensor. So horizontal axis should be ideal, but two vertical axes (optical centers) possibly should be correspondingly 5mm wider from center. But if I will took lenses out, i'll probably mess everything again and will not be able to put it in the right place. I already destroyed one pair of glasses :) But they was of some irregular form and it was extremely hard to find right spot. However my new ones (BTW bought in Auchan for $1.9 - really nice spot where to search for this kind of supply, also the same are in METRO) are of rectangular form, so maybe it will be easier. Will try it tomorrow. Now playing with 100UM, your calibration method and without glasses. As of now I see almost the same level of details and virtually no noise and distortions. Ahhh, what a wonderful workflow after terrible glasses experience :) 

Anyways thanks for timely help. Really appreciate this.

[ck.bar...@gmail.com]

Hey MagWeb,

I've got another round of questions for you.

1. Can I edit the units value in ReconstructMe Console? Is this exclusive to ReconstructMe QT? Is this compatible with OpenNI 1.5 and OpenNI 2?

2. In the reconstructme console --calibrate function, where does it pull the "Factory Depth Intrinsics data" from? Have you ever attempted to update this value to iterate towards a stable value instead of just using the offset?

3. The Carmine advertises a scanning range of 0.35-1.4m. By adding the +2.5 diopter reading glasses, how does this affect the upper and lower limits of the scanning range?

4. Could you continue to increase the "magnification" of the optics and compensate with the --calibrate function? Is there some upper limit where things get crazy?

5. Have you ever investigated splitting a raw data stream from the record function to be used in the multiscan and stitch?

6. Have you noticed any between OpenNI 1.5 and OpenNI 2?

[MagWeb]

Am Freitag, 10. Mai 2013 01:22:00 UTC+2 schrieb ck.bar...@gmail.com:

1. Can I edit the units value in ReconstructMe Console? Is this exclusive to ReconstructMe QT? Is this compatible with OpenNI 1.5 and OpenNI 2?

ReMeConsole still needs OpenNi1.x which does not support submillimeter units. So also ReMeConsole doesn´t have an option to do that.

 

2. In the reconstructme console --calibrate function, where does it pull the "Factory Depth Intrinsics data" from? Have you ever attempted to update this value to iterate towards a stable value instead of just using the offset?

It seems to me me  depthimage stream is created on the sensor using some fix parameters as focal length, intersection angle between sensor and emitter aso. Factory Settings are stored on the devices' firmware. I see no way to modify the depth calculation on the sensor . But if there are data, you can interpret them differently...

3. The Carmine advertises a scanning range of 0.35-1.4m. By adding the +2.5 diopter reading glasses, how does this affect the upper and lower limits of the scanning range?

sure and that's the main effect: Carmine 1.09 is optimized  to scan in that range, but the actual depth values cover a depth up to 7m (about). +2.5 reading glasses condense this total range to about 27cm to 80cm.

BTW: Reading glasses are no "normal" uniform lenses. If you deal with normal optical lenses you may notice some strange thing: To avoid distortion they must be mounted at a certain angle (more about this when I'm through)

4. Could you continue to increase the "magnification" of the optics and compensate with the --calibrate function? Is there some upper limit where things get crazy?

As written above, the either effect of lenses seems to be the "depth data condensation". The zooming effect isn't that much. +3 glasses work too - but the resulting depth range drops to about 20cm. Lenses above +3 do not give data here any more. In my eyes 2.5 lenses are the way to go.

IMO trying to get a closer range would need to modify the sensor's and projector's distance. 

5. Have you ever investigated splitting a raw data stream from the record function to be used in the multiscan and stitch?

Yes I did. Without success - I tried to combine --multiscan + --replay in in a single command. Seems multiscan is dominant to replay

6. Have you noticed any between OpenNI 1.5 and OpenNI 2?

For example : Support of submillimeter units...

[Mark Schafer]

4. Could you continue to increase the "magnification" of the optics and compensate with the --calibrate function? Is there some upper limit where things get crazy?

As written above, the either effect of lenses seems to be the "depth data condensation". The zooming effect isn't that much. +3 glasses work too - but the resulting depth range drops to about 20cm. Lenses above +3 do not give data here any more. In my eyes 2.5 lenses are the way to go.

IMO trying to get a closer range would need to modify the sensor's and projector's distance. 

IMHO to get closer detail we would need to modify the sensor to bring the projector closer to the sensor.
The current math relies on the sensor and projector being exactly level with each other. I looked into modifying the kinect directly but its too hard as neither sensor nor projector is easily moved.
However its possible to create an offset prism using two porro prisms (e.g. from one axis of a binocular). This would allow you to move the sensor camera closer to the projector. Its basically a horizontal adjustment. The distance needed is quite small. Probably only 5-10mm is all that is required.

The problem with a double porro prism is that the image is rotated 180 degrees. So you probably want two sets of double porro prisms to correct for that. An entire set of optics from a pair of binoculars. Or a porro and a porro-abbey to inline rotate.
http://en.wikipedia.org/wiki/Porro_prism
http://osa.magnet.fsu.edu/terms/prisms.html
surplus porro abbey prism from here: http://www.sciplus.com/p/PORRO-ABBEY-PRISMS_35453
They have some 10x50 Binoculars reasonably cheap too. (if you are in USA)

At edmund scientific you get many more choices.
http://www.edmundoptics.com/optics/prisms/
http://www.edmundoptics.com/optics/prisms/specialty-prisms/experimental-grade-porro-prisms/3562
http://www.edmundoptics.com/optics/prisms/specialty-prisms/experimental-grade-specialty-prisms/3561

YMMV

[Mark Schafer]

When I say level with each other, I mean parrallel to each other.
The projector's plane must lie parallel to the camera plane.

[MagWeb]

I tried to add prism optics and some other optical adds to modify the way of the beams:

Some difficulties:

- very high accuracy needed;

- at a closer distance the sensor gets oversaturated (needs a dimming down filter)

- projection and sensor's angle of view  is wide angled which needs rather big reflecting/fracting surfaces in prism or mirror  optics (maybe bigger than the available space)

- the same addon has to be done for sensor and projector t keep them on the same plane in Z

[Mark Schafer]

Yes the size of the prism faces would have to be quite large to cope with the diverging nature of the projector and wide angle camera :-(
I did not think they would have to be in the same Z plane. Just parallel and level "center to center" but I take your word for it, of course.
If that's the case then if we could use the double porro prism from binoculars - this would suggest using one prism over each sensor.
Update - alas porro prism of this size do not seem obtainable :-(

It still looks like the Carmine sensor and MagWeb's +2.5 reading glasses technique are the best bet today.
Pity we can't get the projector a little closer to the camera sensor and thereby decrease the baseline and increase res in small volumes. :-(

[ck.bar...@gmail.com]

Sometime this week I plan to open up my Carmine 1.09 and see how difficult it would be to relocate senors. I can't seem to find a good tear-down so I'll be sure to take pictures. I have some friends in the Optical science department at the University here so I will consult them about the issues you raised. They may also have access to some equipment and optics that would be prohibitively expensive otherwise.

[MagWeb]

Comparing the Xtion's housing to the Carmine's it is exactly the same. Also the weight of both devices is equal. So I guess their inner constructions are pretty similar too.

Here is a teardown of the Xtion:

http://www.youtube.com/watch?v=mn80jJ4Io04

Primesense's Carmine hardware agreement forbids opening.

Maybe one could use two simple plane pieces of glass.

This gives a parallel translation depending on the glass thickness. (see att)

[Mark Schafer]

The useful Teardown video shows us the entire system is mounted on one circuit board. Alas it does not look good for relocating sensors closer together.

I really like your glass refraction method but I think it will be very difficult to avoid the internal reflections at IR wavelengths :-(.
However the simple approach is often best and your idea is good. To avoid refraction effects we could use a first surface mirror. Its easy to cut (its generally very thin).
It works exactly the same as the prism does but the mirror is placed at each of the 45 degree surfaces in the prism diagram.
Because we can make the mirror surface quite large - should avoid clipping the wide angle effects of camera and projector.

I'm off to the garage...

[Mark Schafer]

Some useful links about this:
Mirror periscope and first surface mirrors:  http://www.physicsatweb.com/rev.php?id=Light&v=true
simple manual lineup mechanism :-)   http://www.t-spy.com/Periscope.html

[ck.bar...@gmail.com]

There are some first surface mirrors at www.surplusshed.com and http://www.goldmine-elec.com/ for very cheap. I am worried about the FOV of the senor and emitter. Without having done a geometry check, I fear it will be difficult to get all those mirrors to "fit" without cutting off one of the edges. Do we know the maximum effective angle that the perimeter IR grid is projected at?