creating panoramas directly from RAW images

[kfj]

Dear all!

In my last post I have already hinted at new possibilities arising from the use of OpenImageIO (OIIO for short). I found that it's quite feasible to use OIIO's image importing code as a stand-in for vigra's importImage. Now I've gone one step further and refitted some hugin tools with OIIO code: pto_gen and cpfind. For pto_gen, I have just changed the retrieval of image metrics to use OIIO, but in cpfind I have also changed the image import code. With the modified tools, I could run a tool chain from a set of RAW images to a readily-optimized PTO file by using first the modified pto_gen, then the modified cpfind and finally autooptimiser. The resulting PTO could be loaded into my recent build of lux (the build from the oiio branch, for which I offered an AppImage in my last post) which also employs OIIO and can process RAW images as input. lux stitched this PTO into a seamless panorama.

All of this was possible without any intermediate TIFF files on disk - OIIO handles the loading and processing of RAW images in-library using libRAW and presents the readily-converted data. I think with this little experiment I have established that OIIO would make a good candidate to replace vigra's ageing impex library - OIIO can also handle newer formats like HEIF and webp, among many others. Since the swapping-in of OIIO code for vigra code is quite painless, one might assume that the entire hugin software collection could be refitted like this, bringing hugin up-to-date with newer file formats and adding RAW support.

[David W. Jones]

And how does it handle the need for noise reduction and sharpening in
RAW files?

--
David W. Jones
gnome...@gmail.com
wandering the landscape of god
http://dancingtreefrog.com
My password is the last 8 digits of π.

[dudek53]

This would be nice, to have align_image_stack and enfuse chewing raw files instead of intermediates.

--
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
---
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hugin-ptx+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hugin-ptx/cf611e0c-978f-4516-b3ad-8c0396036f3dn%40googlegroups.com.

[David W. Jones]

That would include chewing on the raw digital noise in RAW files, and lacking the sharpening that said files most likely need.

Hugin support for EXR files would be great.

On 2/22/24 11:13, dudek53 wrote:

This would be nice, to have align_image_stack and enfuse chewing raw files instead of intermediates.

On Thu, Feb 22, 2024 at 6:37 PM 'kfj' via hugin and other free panoramic software <hugi...@googlegroups.com> wrote:

Dear all!

In my last post I have already hinted at new possibilities arising from the use of OpenImageIO (OIIO for short). I found that it's quite feasible to use OIIO's image importing code as a stand-in for vigra's importImage. Now I've gone one step further and refitted some hugin tools with OIIO code: pto_gen and cpfind. For pto_gen, I have just changed the retrieval of image metrics to use OIIO, but in cpfind I have also changed the image import code. With the modified tools, I could run a tool chain from a set of RAW images to a readily-optimized PTO file by using first the modified pto_gen, then the modified cpfind and finally autooptimiser. The resulting PTO could be loaded into my recent build of lux (the build from the oiio branch, for which I offered an AppImage in my last post) which also employs OIIO and can process RAW images as input. lux stitched this PTO into a seamless panorama.

All of this was possible without any intermediate TIFF files on disk - OIIO handles the loading and processing of RAW images in-library using libRAW and presents the readily-converted data. I think with this little experiment I have established that OIIO would make a good candidate to replace vigra's ageing impex library - OIIO can also handle newer formats like HEIF and webp, among many others. Since the swapping-in of OIIO code for vigra code is quite painless, one might assume that the entire hugin software collection could be refitted like this, bringing hugin up-to-date with newer file formats and adding RAW support.

-- 
David W. Jones

[kfj]

> And how does it handle the need for noise reduction and sharpening in RAW files?

Processing the RAW file is done with configuration parameters, please refer to the OIIO docu for available options - among them noise reduction which is best done early in processing. Sharpening is best done on the finished panorama.

[dudek53]

It seems you came far into this libraw implementation. Would be of big interest to have this working with enfuse or align_image_stack. How would it handle tonemapping code though 👀?

Den fre 23 feb. 2024 08:04'kfj' via hugin and other free panoramic software <hugi...@googlegroups.com> skrev:

> And how does it handle the need for noise reduction and sharpening in RAW files?

Processing the RAW file is done with configuration parameters, please refer to the OIIO docu for available options - among them noise reduction which is best done early in processing. Sharpening is best done on the finished panorama.

--
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
---
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hugin-ptx+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/hugin-ptx/8a6dade3-1633-4d34-9d84-c1368c9e95adn%40googlegroups.com.

[David W. Jones]

On February 22, 2024 9:04:18 PM HST, 'kfj' via hugin and other free panoramic software <hugi...@googlegroups.com> wrote:
> > And how does it handle the need for noise reduction and sharpening in RAW
> files?
>
> Processing the RAW file is done with configuration parameters, please refer
> to the OIIO docu

> <https://openimageio.readthedocs.io/en/v2.5.8.0/builtinplugins.html#raw-digital-camera-files>

> for available options - among them noise reduction which is best done early
> in processing. Sharpening is best done on the finished panorama.
>

Ah. I prefer interactive ways of setting noise reduction. I then feed the processed TIFFs into Hugin, output panoramas as EXR files, and use those for final processing into JPGs.

--
David W. Jones
gnome...@gmail.com

exploring the landscape of god
http://dancingtreefrog.com

Sent from my Android device with F/LOSS K-9 Mail.

[kfj]

> It seems you came far into this libraw implementation. Would be of big interest to have this working with enfuse or align_image_stack. How would it handle tonemapping code though?

My proposal is simply about reading image files into the program. it's to replace use of libvigraimpex for image import, that's all. Everything else remains just the same. This is why it was so easy to 'slot it in' to cpfind and pto_gen: I just looked for vigra::ImageImportInfo and replaced it with a struct which uses OIIO data internally, then I looked for vigra::importImage and replaced it with calls to an equivalent routine using OIIO. Apart from a bit of glue code to mimick the vigra::ImageImportInfo structure and to map parameters originally passed to vigra::importImage to the code using OIIO there were no changes. I already had the glue code, because I use it in lux.

Tonemapping etc. is not in the scope of this proposal. That's the beauty of it: all code can remain as it is, only image import is re-routed to a powerful, modern library with lots of interesting features. The glue code resides in a small pair of .cc and .h files, and inside the code being transformed, all it takes is an include statement and the replacement of vigra::ImageImportInfo and vigra::importImage with calls into the glue code. It's minimally invasive. The down side is that OIIO is a large library. But it's very available - I found it on all platforms lux supports - various linux distros, macports, mingw and freeBSD.

[dudek53]

That sounds very interesting. Mac user here. I probably won't be able to implement this myself. Would it be possible for you to have some code to be able and compile for Mac?

--
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
---
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hugin-ptx+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/hugin-ptx/c4909668-09ce-4f8e-8846-a64ff81b3702n%40googlegroups.com.

[kfj]

On Friday, February 23, 2024 at 8:47:07 AM UTC+1 dud...@gmail.com wrote:

That sounds very interesting. Mac user here. I probably won't be able to implement this myself. Would it be possible for you to have some code to be able and compile for Mac?

I'm just brushing up the glue code, then I'll publish it. All you need is to ad a .cc file to your source files, include a header into the files which you want to transform and replace the info structure and the calls to vigra::importImage read the data. You also have to up the C++ standard to C++14 for OIIO and link to OIIO. If you have a working build environment, these changes are not hard to do. When I find the time I can see if I can get it to work on my mac - Routinely I only build lux on that mac, but maybe building hugin and friends is not too hard - after all macOS is unixoid. No promise, though.

[dudek53]

Thanks a lot for sharing hope on Mac side ;). I'll have a look into what you are describing when time is on my hands again. I'll be back with more questions I'm sure 😎.

--

A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
---
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hugin-ptx+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/hugin-ptx/7c0139e1-d572-4827-9e0e-45b058a57a3dn%40googlegroups.com.

[kfj]

Okay, here goes: the code is online
hugin developers, I can offer a patch against hugin master if you're interested in trying it out.

[dudek53]

Thank you!

I wouldn´t mind testing a patched version.

On Fri, Feb 23, 2024 at 9:30 AM 'kfj' via hugin and other free panoramic software <hugi...@googlegroups.com> wrote:

Okay, here goes: the code is online
hugin developers, I can offer a patch against hugin master if you're interested in trying it out.

--
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
---
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hugin-ptx+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/hugin-ptx/2d44ac58-76ca-4236-b19f-ca7bde3c1259n%40googlegroups.com.

[Bruno Postle]

On Fri, 23 Feb 2024, 08:30 'kfj' via hugin and other free panoramic software wrote:

Okay, here goes: the code is online
hugin developers, I can offer a patch against hugin master if you're interested in trying it out.

This looks like something that could be #ifdef'd so it can be a Hugin build option. Though the UI would need some changes to allow heic files in the file picker etc..

-- 

Bruno

[kfj]

On Friday, February 23, 2024 at 10:54:38 AM UTC+1 bruno...@gmail.com wrote:

This looks like something that could be #ifdef'd so it can be a Hugin build option.

This is roughly what I am aiming at. My call signatures are more complex, because the OIIO code does not interface with vigra data containers directly but takes explicit separate stride arguments. But this could be hidden with more go-between-code - as stated, I wanted a quick proof-of-concept, mainly to see if I could get the entire toolchain set up to work from a bunch of RAW files all the way to a stitched panorama. If there is interest on the hugin side to take this further, I'm willing to code some more and make it easy for hugin to try this out for the hugin code base by supplying import functions with adapted signature.

Though the UI would need some changes to allow heic files in the file picker etc

I also had the UI issue when switching to use OIIO in lux. My old code offered a file-select with common image file extensions, but this became impractical with the large number of image file formats OIIO supports, so I now simply offer all files for opening and silently ignore those files in the selection which don't check out as OIIO-compatible. OIIO works like vigraimpex: it looks into the files and opens them with the appropriate plugin, no matter what the extension is. I wrote a replacement for vígra's isImage as well which helps with figuring out whether a file can be used as image input or not, and I could add that to the oiio_fileio repo.

If we could get it to the point where all it would take is to replace the type of the info structure and the name of the import routines, making it a compile option should work out, and the new functionality would become available throughout the hugin tool chain. hugin does try and open, e.g., panoramas with .CR2 files, but fails miserably because libvigraimpex doesn't understand them properly - I think libvigraimpex interprets them as TIFFs and it all comes out wrong.

[dudek53]

I am all in for porting to whole Hugin program.

--

A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
---
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hugin-ptx+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/hugin-ptx/bed876ff-daaa-4ab6-bd6c-b2a2b2456894n%40googlegroups.com.

[kfj]

On Friday, February 23, 2024 at 11:31:40 AM UTC+1 dud...@gmail.com wrote:

I am all in for porting to whole Hugin program.

 

Do you have a working build setup on your mac? then I can send you the patch to try it out. I just tried to see if I could quickly get hugin to build on my mac and I failed right at the beginning, where it failed to find my wxwidgets install from macports. I don't really want to spend much time on getting the hugin build going on my mac, I'd much rather see someone else doing it who already has a working build setup on their mac.

[dudek53]

Yes, I have compiler working for both apple Silicon arm64 and Intel mac 👌

--

A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
---
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hugin-ptx+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/hugin-ptx/9885f833-49c9-4785-a12f-2467a362fd87n%40googlegroups.com.

[kfj]

On Friday, February 23, 2024 at 11:40:09 AM UTC+1 dud...@gmail.com wrote:

Yes, I have compiler working for both apple Silicon arm64 and Intel mac

Excellent. I just pushed the patch to the oiio_fileio repo. On top of applying the patch, you need to put fileio.cc and fileio.h alongside the source code for cpfind (that's in src/hugin_cpfind/cpfind) where the patched code expects it. As I said, this is just a quick shot, and it may not even work on the mac. And for now, it's only for pto_gen and cpfind, not the wole of hugin. Don't forget to get OpenImageIO from macports. If you run into problems, I'll try and help as best as I can.

[dudek53]

Very nice. Did a git pull. I will dig into this probably tonight. Work over here soon. 

Exciting!

--

A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
---
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hugin-ptx+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/hugin-ptx/d6d26223-813e-4ab5-8e50-617f8f90a27en%40googlegroups.com.

[kfj]

On Friday, February 23, 2024 at 11:54:20 AM UTC+1 dud...@gmail.com wrote:

Very nice. Did a git pull. I will dig into this probably tonight. Work over here soon. 

Exciting!

 :D

And if you want to stitch the panorama conatining e.g. RAW files, try a lux build from the oiio branch. I tried that here and it works fine, I just haven't published .dmg on the lux download page because of some licensing issues I want to resolve first.

[T. Modes]

kfj schrieb am Freitag, 23. Februar 2024 um 08:35:28 UTC+1:

My proposal is simply about reading image files into the program. it's to replace use of libvigraimpex for image import, that's all. Everything else remains just the same. This is why it was so easy to 'slot it in' to cpfind and pto_gen:

Sorry for kill the joy. But when taking the full Hugin workflow into account and take all advantage of the raw format it becomes fast more complicated.

It's not so easy to just replace the load routines, the whole ecosystem around have to be adopted.

Just to mention some things to considers:

* The colorspace of the image data has to controlled, ideally a linear colorspace with a wide gamut and not default gamma corrected sRGB of OIIO.

* Disable all automatic changes/adoption by libraw (auto brightness, …).

* Control the white balance, so that all images get the same wb.  (e.g. reading from the first one and apply this then to the other images)

* Take care of the cropping of the raw images (they often contain black borders).

* Take care of rotation of the images.

* Allow the user to change some parameters (debayer algorithm, noise, …). Then the user want a GUI to change them interactive and fast preview. These would mean to implement a crude raw converter GUI into Hugin.(Just out of curiosity, OIIOs raw import has at least 24! different user settings for the raw import.)

These are only the first issues came me into consideration - these are simply ignored by your proposal. 

(There may be some program parts which would also work without take all this considerations. But it needs to be taken into consideration the whole workflow.)

Taking all together would mean to extend the file format to save the new settings and the load routine would needs to update to take care of them.

Converting a raw image takes also some run time. How does this affect the responsivity of the GUI, especially when loading several images or when the caching of the images is needed?

So it's not done with simply replacing the load image call with another library. 

The whole ecosystems needs to be adopted and you have to implement a lot of functions of a raw converter.

IMHO there are dedicated raw converter better suited for these task. Hugin can already take care of some of them. This I consider a better solution, than to open Pandoras box and implement a lot of RAW converter features into Hugin.

[kfj]

On Friday, February 23, 2024 at 5:01:45 PM UTC+1 T. Modes wrote:

when taking the full Hugin workflow into account and take all advantage of the raw format it becomes fast more complicated.

...

These are only the first issues came me into consideration - these are simply ignored by your proposal. 

 

I am offering a proof of concept to demonstrate what's possible. But of course your concerns are valid. As I already pointed out  to Gnome Nomad, OIIO's raw plugin is highly configurable. It offers roughly the same functionality as libRAW upon which it is based, which in turn offers roughly the same functionality as dcraw. OIIO is quite good at taking in parameters from outside. In lux, I simply pass parameters to OIIO and it's plugins through as strings, and OIIO parses them and passes the plugin-specific ones on to the plugin.

Look at it in a different way: OIIO opens TIFFs just fine. There's noone stopping you from using the same workflow as ever, but you *could* also try and work from RAW directly. It's a new possibility to play with, to see if it opens up avenues which weren't possible before. It's an experiment. You're sure not killing the joy, and if you insist on keeping hugin the way it has always been - go ahead. My proposal - using a toolchain with modified pto_gen and cpfind, then autoopitimser and finally lux to stitch - doesn't even use hugin - it was dud...@gmail.com who proposed changing all of hugin.

Modifying cpfind was the missing link for me to get a toolchain together which works in RAW entirely. A lot of my photography is in RAW, and I throw away the TIFFs after stitching because they take up a lot of space. With the oiio branch of lux I can now simply replace .tiff with .CR2 in the PTOs and stitch from RAW, because I used dcraw in the first place. And for my Samyang fisheye, I can even pass through chromatic aberration parameters... I find that a remarkable achievement.

So it's not done with simply replacing the load image call with another library. 

The whole ecosystems needs to be adopted and you have to implement a lot of functions of a raw converter.

 

No, you don't. They are already all there, and you simply pass the parameters you want. Again, here's the link to OIIO's libRAW plugin , you'll see it's pretty much all there. And in lux you can simply pass these args on e.g. on the command line, like --oiio_arg=raw:user_flip=0

IMHO there are dedicated raw converter better suited for these task. Hugin can already take care of some of them. This I consider a better solution, than to open Pandoras box and implement a lot of RAW converter features into Hugin.

 

What the better solution is will show in time. I find a quick and easy option to work directly from RAW is a bonus, even if it takes some extra time to decode the images. If any of this turns out to be a good solution to adopt throughout the hugin codebase can't be decided now, but I wanted to show a path forward. Or will you go ahead and implement vigraimpex import plugins for e.g. HEIF and WEBP, to name but a few? Lux uses a fair bit of vigra, but I feel stuck now that nothing is happening with vigra anymore. My proposal is also about opening up an avenue to a large number of modern image file formats which vigra can't handle because noone there kept abreast of new developments.

[dudek53]

Sounds promising. Points are valid but as an open source spin off there could by forceful synergies in a wip branch.

Let's see where it lands eventually.

--

A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
---
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hugin-ptx+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/hugin-ptx/837730dc-542c-4405-b8a4-5ad9eb171825n%40googlegroups.com.

[T. Modes]

kfj schrieb am Freitag, 23. Februar 2024 um 17:38:09 UTC+1:

So it's not done with simply replacing the load image call with another library. 

The whole ecosystems needs to be adopted and you have to implement a lot of functions of a raw converter.

 

No, you don't.

Sorry, but you are ignoring Hugin photometric optimizer and the photometric corrections done at stitching totally. It may not be import for you, but I see this an important feature of Hugin and this will not work with your proposed workflow..

 

They are already all there, and you simply pass the parameters you want. Again, here's the link to OIIO's libRAW plugin , you'll see it's pretty much all there. And in lux you can simply pass these args on e.g. on the command line, like --oiio_arg=raw:user_flip=0

Sorry, but this something for programmers or nerds. This is nothing a regular user does.

[kfj]

On Friday, February 23, 2024 at 5:56:43 PM UTC+1 T. Modes wrote:

you are ignoring Hugin photometric optimizer and the photometric corrections done at stitching totally.

 

In lux, I only use part of them. I don't use the EMOR and work from linear RGB always. I do use  vignetting information and individual image brightness (Ev) from the PTO - the latter is kind of redundant because lux has it's own 'ligh-balancing' code as well (press Shift+L). I skipped the photometric optimization from my sample workflow with the modified cpfind and pto_gen because it caused an issue here, and I wanted to keep my POC simple.

 

They are already all there, and you simply pass the parameters you want. Again, here's the link to OIIO's libRAW plugin , you'll see it's pretty much all there. And in lux you can simply pass these args on e.g. on the command line, like --oiio_arg=raw:user_flip=0

Sorry, but this something for programmers or nerds. This is nothing a regular user does.

 

Look at the way the parameters are passed to the RAW converters in hugin as it is. That's not so different. Quite nerdy, if you ask me ;-)

[kfj]

Bruno, I added signatures which are compatible with vigraimpex. Have a look, would that work as a replacement? There may be further need for adapters, because some of the code using vigraimpex is old and not using MultiArrayViews (stuff like vigra::DImage instead) - to patch cpfind, I had to brutalize the result of data() which is const for the image container used there. I know it's just the same, but of course that's not so nice. Works, though.

On Friday, February 23, 2024 at 10:54:38 AM UTC+1 bruno...@gmail.com wrote: