Correcting dual fisheye to equirectangular video.

[Evert Vorster]

Hi there.

This request has a little backstory.
I am the proud owner of a Samsung Gear 360 camera, and also a big advocate of open source software.
What I did not know prior to buying the camera is that it does not do stitching of the video inside the camera.
The raw output from this camera is two 190 degree 1920 x 1920 fisheye images on one 3840 x 1920 canvas.
To be able to use this footage, a Windows-only application will stitch it, or one can stitch the video with an app on your phone.
Both these approaches do not allow for any control over the stitching process, and sticks like a thorn in my side because of the closed-sourceness of it.
There are some very expensive software by Kolor that will also do the stitching, but shelling out 680 Euros for what is a hobby for me is also not an option.

There is a project on github by Ultramango that uses a combination of FFmpeg and a Hugin save file and enblend/enfuse to deconstruct the video into frames, do the circular-fisheye to equirectangular conversion, and then stitching it all back together. I have helped a little in fine-tuning the templates used, and a little on streamlining the process. It's a wonderful use of opensource, and the quality and control over the video is really good.
Unfortunately, since the pixel positions for each image are re-calculated from the lens information in the .pto file, the process is quite slow.

I have been playing around with ffmpeg's remap filter, that moves pixels around in the frames based on a map consisting of two pgm images, which encode y translations and x translations respectively.
With this I am able to get a decent 5fps conversion. The problem that I am having with the remap filter in ffmpeg is that the example program that generates the translation maps assumes a perfect fisheye lens.

What I am hoping to do here is to make a translation map based on a Hugin .pto file rather than a perfect lens.

Is there anybody in the Hugin community that can point me to the right person who might be interested to help me?

Kind regards,
Evert Vorster



For the curious, here are a few links:

The project which uses the saved Hugin template:
https://github.com/ultramango/gear360pano

The description of the remap filter and the program that makes an example maps for use in ffmpeg
https://trac.ffmpeg.org/wiki/RemapFilter

Example footage from before the fine tunings in Ultramango
https://youtu.be/GWAwfO16Q3s

Example footage after the fine tuning
https://youtu.be/TfPBc6LcgPQ
This video also uses multiblend instead of enblend.

Example footage done with ffmpeg's remap filter
https://youtu.be/6kTC9IURlTw
There is still more work to do, vignetting and lens corrections.

The original footage:
https://youtu.be/1udaziKqkFA

[T. Modes]

Am Freitag, 4. August 2017 13:13:47 UTC+2 schrieb Evert Vorster:

I have been playing around with ffmpeg's remap filter, that moves pixels around in the frames based on a map consisting of two pgm images, which encode y translations and x translations respectively.
With this I am able to get a decent 5fps conversion. The problem that I am having with the remap filter in ffmpeg is that the example program that generates the translation maps assumes a perfect fisheye lens.

What I am hoping to do here is to make a translation map based on a Hugin .pto file rather than a perfect lens.

Nona with switch -c should output an coordinate map. Maybe you can use this files to create your translations map.
Or there is the tool pano_trafo which can calculate also the positions.
But both approaches needs some more work to get your translation maps.

[Evert Vorster]

Thank you, T. Modes.

I will look into this.

Kind regards,

Evert

--
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
---
You received this message because you are subscribed to a topic in the Google Groups "hugin and other free panoramic software" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/hugin-ptx/a9HP3-tzn2o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to hugin-ptx+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hugin-ptx/d0f34a3f-dde8-4b9a-8b02-d3fec10d439d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Evert Vorster
Isometrix Acquistion Superchief

[Bruno Postle]

On Fri 04-Aug-2017 at 04:13 -0700, Evert Vorster wrote:
>
> I have been playing around with ffmpeg's remap filter, that moves
> pixels around in the frames based on a map consisting of two pgm
> images, which encode y translations and x translations
> respectively. With this I am able to get a decent 5fps
> conversion. The problem that I am having with the remap filter in
> ffmpeg is that the example program that generates the translation
> maps assumes a perfect fisheye lens.
>
> What I am hoping to do here is to make a translation map based on
> a Hugin .pto file rather than a perfect lens.

Oooo, this is an interesting ffmpeg feature.

Attached is a quick script that generates the two map files from any
PTO project (it requires the perl Panotools::Script module and it
probably only works on a unix-y system due to the way it pipes
pano_trafo).

This is very fast, I can see this being useful for converting
fisheye video to Panini projection for example. A drawback is that
ffmeg only uses integer coordinates, so the sampling quality isn't
so great.

Unfortunately, this script only generates maps for the first image
in the project, so it isn't going to directly work for your
use-case. Also, Panotools::Script doesn't know about PTO masks, so
this will map all the unwanted areas in your input photos 🙁. You
could edit the PGM files in an image editor and white out the areas
you don't want, then overlay two streams in ffmpeg, maybe.

This is an often requested Hugin feature, I think it would be useful
to be able to generate these PGM files directly from Hugin. The
trick for rendering these multiple-fisheye-camera files is to load
the file multiple times in the PTO project, so this would need to be
noticed somehow.

--
Bruno

[Evert Vorster]

Hi there, Bruno.

Since the cameras do not change, all I need is the pgm file pair for each lens.

Creating the .pto file for each lens is a doddle, and if there is a way to generate the corresponding .pgm maps for each lens, I am home free.

Attached is my ffmpeg script. Also attached is the projection program to create the pgm files, as well as the alpha map file that the script uses to blend the files.

Here is how the script works.

It uses the -filter_complex of ffmpeg, and initially splits the video input into two streams, left and right.

Then it uses the remap filter on each stream. Since the remap .pgm files are identical for each lens, in this script it re-uses them. If I am able to get a custom .pgm pair for each lens, modifying the script to use a custom remap for each stream is not difficult.

As you will notice, the re-mapped files are larger than the 1920x1920 input. In this case 2060x2060.

The next filter then crops the top and bottom off, leaving the sides for a bit of overlap. This is possible because the Gear 360 is actually 190 degrees per lens.

I then apply an alpha map to the left hand stream, to allow for simple blending. Then, I split the left hand stream in two again.

I then apply the right hand lens in the middle of the original input canvas, and overlay the right part of the left hand split on the extreme left, and the left hand part of the left lens on the extreme right, creating two blends at 25% and 75%. The alpa map can be modified with Gimp to give a bigger or smaller blend.

Once all the fine tuning is done, I don't expect to change either the remap or alpha files, as the two lenses of the camera is fixed in respect to each other.

The example files are zipped, as each of the pgm files are 18mb. Fortunately they compress well.

Kind regards,

Evert

--
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
--- You received this message because you are subscribed to a topic in the Google Groups "hugin and other free panoramic software" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/hugin-ptx/a9HP3-tzn2o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to hugin-ptx+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/hugin-ptx/20170805083353.GA3162%40postle.net.

For more options, visit https://groups.google.com/d/optout.

[T. Modes]

Hi Bruno

Am Samstag, 5. August 2017 10:33:58 UTC+2 schrieb Bruno Postle:

This is an often requested Hugin feature, I think it would be useful
to be able to generate these PGM files directly from Hugin.  

Shouldn't nona -c create these files? Nona -c creates 2 TIFF files with the x and y coordinates of the input images. This should also take masks into account and can also create files for several images.
What's the problem with this approach? (The TIFF files could be converted with ImageMagick to pgm, maybe in an user defined output sequence.)
 
Thomas

[Evert Vorster]

Hi there, T. Modes & Bruno.

Thank you both for the input.

I used Bruno's little script, it avoids having to do a conversion. It does, however, add a new dependancy.

The output video is now as near perfect as I can get.

Kind regards.

Evert

--

A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
---
You received this message because you are subscribed to a topic in the Google Groups "hugin and other free panoramic software" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/hugin-ptx/a9HP3-tzn2o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to hugin-ptx+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/hugin-ptx/2ca846f9-b68e-4336-afa4-203c45385980%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Bruno Postle]

On 5 August 2017 at 10:55, T. Modes wrote:
>
> Shouldn't nona -c create these files? Nona -c creates 2 TIFF files with the
> x and y coordinates of the input images. This should also take masks into
> account and can also create files for several images.
> What's the problem with this approach? (The TIFF files could be converted
> with ImageMagick to pgm, maybe in an user defined output sequence.)

I didn't investigate these nona -c TIFF files. So I have no idea if
they are a similar format to the PGM files needed by ffmpeg. Do they
have all the masking information? or even the seam generated by the
internal blender? if so this should work well, converting from TIFF to
PGM is easy.

--
Bruno

[Carlos Eduardo G. Carvalho (Cartola)]

Sorry for the delay, but have you guys seen this solution: https://github.com/ultramango/gear360pano ?

Cheers,

Carlos E G Carvalho (Cartola)
http://cartola.org/360
http://www.panoforum.com.br/

--
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
---

You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hugin-ptx+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hugin-ptx/CAJV99ZgGA0DQ8QvOSyt7SRXEOp%3DWtkLYBqufScBR52hrbu5LbA%40mail.gmail.com.

[Evert Vorster]

Hi there!

Thank you very much for the link. Unfortunately, that page was the inspiration for the project that I embarked upon.

I have a properly working solution for me, and my own github page:
https://github.com/evertvorster/dualfisheye2equirectangular_ffmpeg_remap

Of course the script and manual could use a little more refinement, and I will get to that once I have a little time on my hands.

Kind regards,

-Evert-

You received this message because you are subscribed to a topic in the Google Groups "hugin and other free panoramic software" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/hugin-ptx/a9HP3-tzn2o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to hugin-ptx+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hugin-ptx/CALW1f7jS6%3DXKdu-LJL-tBnhqc2ux7hrykyWeYvYajNZnpf6n6A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.