Stereoscopic Panoramas

[Simon Bethke]

Hi,

I am an amateur photographer and lately got a Gear VR device so I wanted to try some VR Stuff.

Testing free panorama tools, I found that Hugin was the only free tool providing decent results. Also, my 'Fisheye' lens uses Stereographic projection, so most other tools are not even able to stitch my input material correctly.

Now, the last days, weeks, ... even months(?) ... I am trying to find a good way to capture stereoscopic panoramas with my one camera. After alot of trial and error, I found that the best method for me is to place the camera left and right the NPP and take 12 images for the 360° circle. Now with hugin, I align them and export the frames but don't stitch them yet. For that I use the multiblend tool and first extract its seams. Then I edit the seams that left and right eye matches and blend with these optimized seams. My problem with this method is, that by taking 12 left-eye images and afterwards 12 right-eye images, the time between the left and right eye is to long: Clouds pass by and even the sun travels. Flipping between left and right is inpractical and causes many mistakes if I try to do it fast.

What I would like to do is to have the camera in front (not left or right) of the NPP, take 12 photos and use only right image-halfes for the left eye and vice versa. This works more or less, but here the big issue is, that the paralaxe gets too strong and the stiching result is a mess with alot of visible seams. At this place I wish it was possible, that hugin looks for areas with bad control-points. If the photos were taken from a tripod (with paralaxe), the error should equal the depth of the control points. At least if there are areas with a bigger error (and similar error vectors) and others with less error, the software could treat this as depth. Now with depth information of the control-points, this becomes something similar to visual sfm. The controlpoints are like a sparse point cloud and I'd suggest simply interpolating the areas between the controlpoints to get a perfect stitching result.

If hugin could actually support stitching photos that are taken off the NPP. It could also assist creating stereoscopic panoramas.

What do you think about this? Is Hugin still actively developed or is it rather a complete product which is just supported?

Kind Regards,

Simon

[Carl von Einem]

Hi Simon,

Wim once posted a nice list of notes for his talk at the panotools
meeting 2011 in Vienna: <http://www.panotools-meeting.com/node/104>
where he also links to a very nice and helpful tutorial he wrote:
<http://www.nodalninja.com/forum/showthread.php?4114-Revised-version-of-the-tutorial-quot-How-to-make-a-spherical-3D-panorama-with-a-single-camera-and-a-fisheye-lens>.
Wim uses PTGui but you should be able to follow his tutorial using hugin.

Carl

Simon Bethke wrote on 22.04.16 00:14:

> Hi,
> I am an amateur photographer and lately got a Gear VR device so I wanted
> to try some VR Stuff.
> Testing free panorama tools, I found that Hugin was the only free tool
> providing decent results. Also, my 'Fisheye' lens uses Stereographic
> projection, so most other tools are not even able to stitch my input
> material correctly.
>
> Now, the last days, weeks, ... even months(?) ... I am trying to find a
> good way to capture stereoscopic panoramas with my one camera. After
> alot of trial and error, I found that the best method for me is to place
> the camera left and right the NPP and take 12 images for the 360°
> circle. Now with hugin, I align them and export the frames but don't
> stitch them yet. For that I use the multiblend

> <http://horman.net/multiblend/> tool and first extract its seams. Then I

[Sean Greenslade]

On Thu, Apr 21, 2016 at 03:14:21PM -0700, Simon Bethke wrote:
> Hi,
> I am an amateur photographer and lately got a Gear VR device so I wanted to
> try some VR Stuff.
> Testing free panorama tools, I found that Hugin was the only free tool
> providing decent results. Also, my 'Fisheye' lens uses Stereographic
> projection, so most other tools are not even able to stitch my input
> material correctly.
>
> Now, the last days, weeks, ... even months(?) ... I am trying to find a
> good way to capture stereoscopic panoramas with my one camera. After alot
> of trial and error, I found that the best method for me is to place the
> camera left and right the NPP and take 12 images for the 360° circle. Now
> with hugin, I align them and export the frames but don't stitch them yet.

> For that I use the multiblend <http://horman.net/multiblend/> tool and

It's not a shortcoming of Hugin that non-NPP pano sets are hard to
stitch. It's simply the way optics work. There will be parallax errors
if you're not spinning around the NPP, and there's nothing that can be
directly done about that. Parallax errors are extremely difficult for
software to handle automatically. You can of course use Hugin to stitch
non-NPP sets, and even handheld shots. Often times, some clever manual
masking can reduce or even eliminate the more severe parallax errors,
but that relies on there being "fudgable areas," e.g. blank white walls
where you can hide the misaligned seam.

The advice from that forum post in the other reply seems to be pretty
good. Keep the number of photos as low as possible to keep the number of
seams down, and avoid very close objects to the camera (which cause the
most severe parallax).

The problem is that 3D and pano-stitching are essntially at odds with
each other. 3D uses parallax to imply depth, while panos want as little
parallax as possible (ideally zero), since it causes stitching artifacts.

--Sean

[Carlos Eduardo G. Carvalho (Cartola)]

2016-04-22 9:12 GMT-03:00 Sean Greenslade <se...@seangreenslade.com>:

The advice from that forum post in the other reply seems to be pretty
good. Keep the number of photos as low as possible to keep the number of
seams down, and avoid very close objects to the camera (which cause the
most severe parallax).

I haven't done 3D panoramas yet, but from what I've read it looks like you can't do a few number of images. 12 looks ok, as I've already read that 20 would be a good number. There was a post at facebook where people discussed the small number possible. Nobody was able to do with 4 and the results were improved when increasing the number.

Maybe some device like this - http://cdn.instructables.com/FC0/X56G/HFPTJG8S/FC0X56GHFPTJG8S.LARGE.jpg - can reduce the shots in half.

Carlos E G Carvalho (Cartola)
http://cartola.org/360
http://www.panoforum.com.br/

[Simon Bethke]

Interesing link. Unfortunately, everything Wim talks about is stuff, I already found out the hard way ;) Really, he explains exactly the process i do with Hugin without me knowing of his work :D

[Simon Bethke]

I am not saying that non-NPP issues are a shortcomunig of Hugin. I am saying, that morphing the controlpoints in a way a texture is mapped on a wireframe model (using baricentric coordinates in a triangle) will allow stitching non-NPP panoramas without visible seams. The biggest issue then will be occlusion, which will cause half-transparent areas. A workaround for this could be occlusion detection which should be possible by increasing the density of controlpoints in areas that are (partially) not mappable.

[Carl von Einem]

You _already_ found out? Does that say you have examples from before
summer of 2011?

Simon Bethke wrote on 22.04.16 15:24:

[Simon Bethke]

No no no :)

I found that out the last weeks. I just didn't know about his experiences.

[Carl von Einem]

Hi Sean and Carlos,

I followed Wim's tutorial (see my earlier message) and my results looked
great on Oculus Rift. As for the number of images the workflow reminded
me of the ObjectVR shots I did around 1997: for me 36 images are a good
number of frames to process. With 36 steps for one panorama you want to
avoid moving objects though.

In this case I used my circular fisheye (the 2.8/8mm Nikkor) on a full
frame camera, a sturdy tripod and my rock solid panoramic head which has
a 10 deg. setting. No bracketing, just 36 RAW images which took me
exactly 1 min 30 sec to shoot (including two frames I shot twice).

Stitching images with such a vast amount of overlap works like a charm,
even with the parallax "errors" I had provoked.

Carl


Carlos Eduardo G. Carvalho (Cartola) wrote on 22.04.16 15:03:

[Carl von Einem]

I know we have no tutorial (yet) about this technique on
http://hugin.sourceforge.net/tutorials/ and unfortunately no earlier
discussion on this list mentioning his findings.

Simon Bethke wrote on 22.04.16 16:03:

> <http://www.panotools-meeting.com/node/104>
>

[Erik Krause]

Am 22.04.2016 um 15:24 schrieb Simon Bethke:
> I am not saying that non-NPP issues are a shortcomunig of Hugin. I am
> saying, that morphing the controlpoints in a way a texture is mapped on a
> wireframe model (using baricentric coordinates in a triangle) will allow
> stitching non-NPP panoramas without visible seams.

Thomas Sharpless (inventor of the pannini projection) is attempting
something like this:
http://ivrpa.org/news/practical-stereo-stitching-with-pt3d-quebec-2016/

--
Erik Krause
http://www.erik-krause.de

[Tom Sharpless]

It is possible to stitch a 3D panorama from as few as 4 views taken with a stereo pair of cameras.  I routinely shoot them 6 around, never more than 12.  The keys to success are, first, to align the the views 'at infinity' -- that is, to the true direction and roll of the views -- and second to warp the images to fit,  warping the left and right images of each view as nearly the same as possible, so that stereo disparities are preserved in the pano pair.  

Alignment at infinity can be approached in several ways.  Perhaps the best is to use a stable high-precision rig that reproduces camera attitudes to subpixel accuracy, and create a master alignment template used for all stitches.  But that is very expensive and technically demanding.  The same idea can be applied with ordinary equipment if you mount one camera on-nodal and use that image series to generate a template.  But this doubles the error in the other sphere, compared with having the axis halfway between the lens pupils.  Peter Murphy (a master) often adds a third, small on-axis camera to a symmetric pair, for the sole purpose of generating an alignment template.  

Another way is to try to select a set of control points that makes a good alignment template.  This will contain only points on distant parts of the scene.  You can often make it better by combining control points from left and right cameras.  Such a CP set is not so good for the warp-to-fit step, but the goal here is just to align the views correctly.

For warp-to-fit, I use all the CPs linking adjacent images (but not more distant ones) and apply PTGui's viewpoint correction to all images, with the y,p,r alignment locked at the template values.  I'm not current with what Hugin can do along these lines, but I'd guess it may be even more capable.  One could use the template idea here too, by actually combining all left and right CPs and saving out a warp template to be used for the final stitches.

I have developed a helper program for PTGui, called PT3D, that automates this kind of control point fiddling and combines both fine alignment and warp-to-fit in a single step that processes left and right spheres together.  It usually gives alignments that can be stitched successfully if you put in enough effort on blend masks.  If you have PTGui and are interested to try it, it is available under commercial license at http://panini-pro.com/PT3D/ 

At the outset I designed PT3D to be compatible with Hugin as well, but by now it has many specific dependencies on PTGui.  However I am open to the idea of an opensource version specifically for Hugin, if someone wants to develop it.

[Javi Baranano]

Hello Simon, now it's much easier... https://vr3d360.org/