new c++ code for environment images and reprojection

[kfj]

For the past couple of weeks, I've been diving into some new code. It all started out when I started using OpenImageIO as image i/o library for lux. OpenImageIO provides some functionality which is relevant in a panorama programming context, namely it's 'environment' function, which samples a full spherical with proper antialiasing and interpolation. I found this function 'tasty' and started playing with it. This led me to try feeding 'cubemaps' to it, which is another format used to cover an entire 360X180 degree environment. Alas, no luck here, not supported. lux has it's own flavour of cubemap processing, but I'm not entirely happy with it's performance, so next I decided to investigate cubemaps to maybe come up with better code. To have some standard to adhere to, rather than my 'ad-hoc' approach in lux, I turned to openEXR's 'flavour' of cubemap. I started out by using their utility to convert between cubemaps and full sphericals ('exrenvmap') but I wasn't happy with it - it didn't work as I expected. Correspondence with the openEXR people turned out that cubemaps are not used much and their code is twenty years old and probably not entirely correct - and slow. Time to start coding myself, and to use my library zimt to code it all in multithreaded SIMD code for speed!

I first coded a utility to convert between full spherical (a.k.a lat/lon) format and cubemap format. This led to more code dealing with cubemaps - the utility is now quite capable and offers a few - hopefully - interesting features. Then, to work towards transferring my new insights and code to lux, I wrote code to reproject environment images (both full sphericals and cubemaps) to common projections: rectilinear, spherical, cylindrical, fisheye, stereographic and cubemap. This latter code is what is probably most relevant in a panorama programming context, so I'd like to point you to it. It may also be of interest to the libpano project. The code is here. There's a demo program 'stepper.cc' which shows how the code base can be employed to code reprojections very tersely - giving a virtual camera orientation with Euler angles is built-in. I may elaborate the demo program - so far, there is no parameterization with CL arguments, it's just a simple pass-an-environment and get preprojected images demo, but I thought that now is a good time to point you to this new project. The code is extensively commented, and it's all MIT-licensed.

[kfj]

Added 'extract.cc' which extracts an image from a lat/lon or cubemap environemnt. You can pass yaw, pitch and roll of the virtual camera, allowed target projections are "spherical", "cylindrical", "rectilinear", "stereographic", "fisheye" or "cubemap", and you can specify width, height, and hfov of the output. Currently, only bilinear interpolation is used and output is RGB only (no alpha). Also builds with cmake.

To be continued! this should make a nice utility for panorama photography.

On Sunday, May 12, 2024 at 10:34:23 AM UTC+2 kfj wrote:

..- so far, there is no parameterization with CL arguments

[kfj]

To make it easier for you to evaluate the two new programs, I've uploaded a debian package to the lux Downloads page. I built and test-installed this package on debian12 - your mileage on other debian-based distros will vary. Building from source is also quite simple, the build is done with cmake. Both envutil and extract are MIT-licensed, even if the package mangement doesn't comprehend that (mine tells me it's 'proprietary').

I've elaborated 'extract' with options to use different interpolation methods. On top of the default bilinear interpolation, there are now two more methods, each with variations. The base method is picked with the --itp parameter:

    1 - use simple bilinear interpolation directly on the source image
        this is the fastest option, and unless there is a significant scale
        change involved, the output should be 'good enough' for most purposes.

    -1 - use OpenImageIO's 'environment' or 'texture' function for lookup.
         without additional arguments, this will use a sophisticated interpolator
         with good antialiasing.

    -2 - use 'twining' - this is a method which first super-samples and then
         combines several pixels to one output pixel ('binning'). This is my
         own invention. It's quite fast and produces good quality output.
         This method should see community review to compare it with other methods.

The second and third methods can be tweaked further - I made an effort to allow practically the entire range of options OIIO can take, and my own 'twining' filter can also be configured further - for details, please look at my github page. If you just want to try it out quickly, try passing --itp -2 and --twine 3 for ninefold oversampling and a 3X3 box filter for pixel binning.

While using OIIO for lookup tends to be quite slow (especially with the default settings), twining is quite fast unless you pick an overly large box filter. I found that the output of twining is nice and crisp and the antialiasing is good - I think this lookup method has potential and I'd welcome feedback, especially comparisons with other methods.

[kfj]

Still with me?

I just uploaded a debian package of version 0.1.1 of 'envutil'. This is now the sole program for conversion between environments (like full spherical panoramas) and extraction of partial images (nadir caps and such) - the separate 'extract' program ended up being able to do the conversions as well, so I decided to lump it all together into a single utility. I have taken the extraction of partial images one step further: envutil now accepts 'sequence files' - simple text files with four numbers per line, coding for hfov, yaw, pitch and roll of a single image. If you feed such a sequence (--seqfile my.seq), envutil will output a video (specify output like --output xx.mp4) - or a set of separate images (specify output like --output img%03d.jpg) . This is good fun, but you'll want to create the sequence file with a script or so (try a simple loop in a shell script for starters). Detailed documentation is on the github page.

[kfj]

On Saturday, June 8, 2024 at 8:56:53 AM UTC+2 kfj wrote:

I just uploaded a debian package of version 0.1.1 of 'envutil'.

 

... how embarrassing - there was a bug in the package, which I've fixed ('early adopters', please download again if you want correct video output with --itp -2) .

So, just to what your appetite: let's suppose you have a full spherical 'pano.jpg', try this:

    for (( x=0 ; x<360 ; x=x+1 )) ; do echo 90 $x 0 0 ; done > xx.seq

This creates a 'sequence file' - in this case, the only thing that varies from line to line is the second number, the 'yaw' parameter - and it will take values from zero to360, so one full pan around. Then this:

    envutil -v --input pano.jpg --seqfile xx.seq --output pan360.mp4 --width 1920 --height 1080 --itp -2

This will create the video output 'pan360.mp4' - a fullHD H265-encoded 360 degree pan of your panorama with rectilinear output set to 90 degrees hfov. The video won't play everywhere - my Linux box plays it, but VLC doesn't. I'm new to video output, maybe there's someone out there in the community who has programmed with ffmpeg and wants to help? Anyway, you can instead produce 360 separate images like this:

    envutil -v --input Cauri*.jpg --seqfile xx.seq --output pan%03d.jpg --width 1920 --height 1080 --itp -2

(note the changed output parameter) - and then combine the images with ffmpeg on the command line, like this:

    ffmpeg -f image2 -pattern_type glob -framerate 60 -i 'pan*.jpg' -s 1920X1080 -c:v libx265 pan2.mp4

That video plays with VLC. Programming with ffmpeg is quite tricky - it's old C code and I grabbed an example file which was 20 years old and 'bent' it to C++ - going via the single images is the safer option, but of course it takes lots of disk space (you want an SSD for that). But I think this is a cool idea - it could be used to create very high quality artificial tracking shots (if that's the correct term) or zoom sequences from a 360 degree panorama with little programming effort. Of course one would want a companion program to generate the sequence files...

[chaosjug]

Hi,

ffmpg is capable to do this in one go. It's been a while since I did this so I must admit that I don't
completely understand what I did, but this should create a 360° rotation pan of a panorama image:

ffmpeg -loop 1 -i panorama_image.jpg -vf "sendcmd=f=cmd.txt,v360=input=e:output=flat:ih_fov=360:iv_fov=180:h_fov=120:v_fov=88.5:w=1920:h=1080:pitch=-10:reset_rot=true:interp=cube" -t 72 -vcodec libx264 -crf 25 -r 50 -pix_fmt yuv420p output.mov

cmd.txt contains this:
0.0 [expr] v360 yaw '-1*(mod(T/72*360+180,360)-180)'

The cmd.txt basically sets the yaw for v360 video filter based on the time of the frame (T)
(I don't remember why I did it this complicated, I think it had something to do with the fact that
yaw needs to be between -180 and +180 and I wanted to start at 0.)

Some more info can be found here:
https://ffmpeg.org/ffmpeg-filters.html#v360
https://ffmpeg.org/ffmpeg-filters.html#sendcmd_002c-asendcmd

Am Samstag, 8. Juni 2024, 18:04:14 MESZ schrieb 'kfj' via hugin and other free panoramic software:
> On Saturday, June 8, 2024 at 8:56:53 AM UTC+2 kfj wrote:
>
>
> I just uploaded a debian package

> <https://bitbucket.org/kfj/pv/downloads/envutil-0.1.1-Linux.deb> of version

[Kay F. Jahnke]

On 08.06.24 22:51, 'chaosjug' via hugin and other free panoramic

software wrote:
> Hi,
>
> ffmpg is capable to do this in one go. It's been a while since I did this so I must admit that I don't
> completely understand what I did, but this should create a 360° rotation pan of a panorama image:
>
> ffmpeg -loop 1 -i panorama_image.jpg -vf "sendcmd=f=cmd.txt,v360=input=e:output=flat:ih_fov=360:iv_fov=180:h_fov=120:v_fov=88.5:w=1920:h=1080:pitch=-10:reset_rot=true:interp=cube" -t 72 -vcodec libx264 -crf 25 -r 50 -pix_fmt yuv420p output.mov

Hey, cool, I'm always impressed by just how versatile ffmpeg is!
Let me make two points:

- The method to use a sequence file is perfectly general, it can process
any sequence of images taken from the same virtual camera location.
envutil also reprojects to six different projections, so I think it's
got a wider range of applications. The 360 degree pan I suggest is
merely a simple example which is easy to generate with a loop in bash.

- My problem just now is how to pump frames into a video in C++, to
avoid the single-image route. The ffmpeg code I use is twenty years old,
can you maybe suggest something else? It should really be quite simple.
I looked around some, and was quite disappointed not to find a
ready-made piece of code which simply 'eats' C arrays of image data and
combines them into a video. It doesn't have to be ffmpeg - I use that
code because OIIO 'pulls it in' anyway, so I don't have to add another
dependency, but I wouldn' mind that, either. Ultimately, of course, I
hope that OIIO improves their ffmpeg plugin to not only read from but
also write to videos. That would be my favourite outcome.

Currently, the video output from envutil doesn't seem to be good for
many video players, so I suggest using the single-image path if the
video doesn't work. That's to be able to compare the different
anti-aliasing filters in a moving image - I hope that my new approach of
'twining' compares quite well!

[Kay F. Jahnke]

On 08.06.24 22:51, 'chaosjug' via hugin and other free panoramic

software wrote:
>
> Some more info can be found here:
> https://ffmpeg.org/ffmpeg-filters.html#v360
> https://ffmpeg.org/ffmpeg-filters.html#sendcmd_002c-asendcmd

Thanks as well for these links! I've read through the documentation, and
it looks like ffmpeg is extremely capable when it comes to producing a
video from an environment. It knows a lot of output projections, and it
even does cubemaps! That took me quite some programming, and I did not
cover other cubemap formats like 3:2 - only 1:6 and six single images.
Dual fisheye format also turns up - that's another candidate I have in
mind to include in envutil, both as input and output. I did some work
with dual fisheyes in lux after someone sent me some sample images from
a Xiaomi mi sphere camera but didn't take it any further then - just
made a PTO to use the camera's image as input to lux.

I tried your example to produce the pan video here, and it worked, but I
think maybe there's an issue. I get this message:

[vost#0:0/libx264 @ 0x558d9c0040c0] More than 1000 frames duplicated

And the video seems to stagger a bit - like jump in small increments
from one view to the next, rather than panning smoothly. Maybe the yaw
angle is calculated in integer arithmetic instead of floating point?
Just a guess. I've played a bit with the cmd.txt, but I can't seem to
get rid of the duplicate frames.

[kfj]

On Monday, June 10, 2024 at 10:09:23 AM UTC+2 kfj wrote:

I think maybe there's an issue. I get this message:

[vost#0:0/libx264 @ 0x558d9c0040c0] More than 1000 frames duplicated

 

My mistake - I tried to generate the video with 60 fps, and the default for the (single-image) input seems to be 25 fps.

The ffmpeg invocation you gave is fine and produces no errors. The video stuttering a bit may be due to my old hardware, or due to running a 25 fps video on a 60 Hz screen.