caught exception: std::bad_alloc with builtin blender

[Caetano Veyssières]

When I use the built-in blender on a big panorama (4 gigapixel), the stitching stops right away and returns this error at the end of the log :

caught exception: std::bad_alloc

• If I divide the resolution by 2, it works fine
• If I use Enblend, it processes for a long time until it stops and returns :
  enblend: out of memory
  enblend: std::bad_alloc

The built-in blender gives better results in my case, so I'd like to solve the problem on this one.

specs :
Hugin 2016 2.0
Linux Mint 18.0 x64
Swap partition : 8 GB
Swap file : 12 GB

Tell me if you need more info
Thanks in advance

[T. Modes]

Am Donnerstag, 15. Dezember 2016 16:37:42 UTC+1 schrieb Caetano Veyssières:

• If I divide the resolution by 2, it works fine
• If I use Enblend, it processes for a long time until it stops and returns :
  enblend: out of memory
  enblend: std::bad_alloc

From the enblend documentation, chapter 2, Known limitation:
Enblend has its limitations. Some of them are inherent to the programs proper others are imported by using libraries as for example VIGRA. Here are
some of the known ones.
* Total size of any even intermediate image is limited to 2^31 pixels, this is two gigapixels.

The same limitation applies to Hugin, because it use the same underlying VIGRA library.
So this explains the bad_alloc.

Which VIGRA version are you using? Not sure if current VIGRA version 11.0 can handle bigger images.

[Caetano Veyssières]

I didn't know about VIGRA. I think the package is named "Libvigraimpex5v5", so I would have version 1.10.0. Should I upgrade ?

[T. Modes]

Am Donnerstag, 15. Dezember 2016 21:23:52 UTC+1 schrieb Caetano Veyssières:

I didn't know about VIGRA. I think the package is named "Libvigraimpex5v5", so I would have version 1.10.0. Should I upgrade ?

Simply updating the library is not enough. Hugin and enblend needs to compiled with the newer version (vigra is mainly a header only library, only a small part of the functions is used from a dynamic link library).
But I can't guarantee that it works with the version 1.11.

[cspiel]

On Thursday, 15 December 2016 16:37:42 UTC+1, Caetano Veyssières wrote:

If I divide the resolution by 2, it works fine

Easy answer: Stick with half the problem size.
 

• If I use Enblend, it processes for a long time until it stops and returns :
  enblend: out of memory
  enblend: std::bad_alloc

    The error message is clear: The Enblend process has not
enough memory to run to completion at the given problem size.

Here are increasingly more difficult possible solutions all
assuming you have not hit the 2^31 pixel limit that Thomas
mentioned.

(1) Check your process limits, i.e. compare `ulimit -d -m -v' with
    `ulimit -d -m -v -H'; adjust soft limits if promising/necessary.
(2) Move to a system with more core memory and/or swap.
(3) [Hard!] Compile the `mmap_view' branch of Enblend/Enfuse on your
    machine and see if it helps, though there are no guarantees.  See
        http://hg.code.sf.net/p/enblend/code/changeset/7a3964af671a
    In theory `mmap_view' should decouple Enblend/Enfuse from the
    implictly used VMM and take over mapping memory for large
    images.

HTH,
    Chris

[Caetano Veyssières]

Thanks. I think I will do a fresh Linux install with more swap (I need to do a fresh install anyway).
But are we sure Hugin uses swap files ? If I have an 8GB swap partition and a 12 GB file, will it work the same as having a 20 GB partition ?

[Gnome Nomad]

Swap space is treated as memory. Temporary files, such as ones created in /tmp, are NOT memory. They're ordinary disk files. My present laptop has ~420GB of free space, so if something needs to create a 400GB temp file, no problem!

Linux uses swap space whenever a typical program needs more memory than what's available. So if Hugin needs more, Linux will use swap space.

I used to use Hugin on a 32-bit Linux system with only 2GB of memory. It happily used swap space when processing big images. Took a long time - but it did it.

You don't need to reinstall to enlarge a swap partition. Back up your system (& home partitions if separate). Boot from a Live linux distro like Knoppix or Ubuntu. Use Gparted to shrink another partition enough for the space you want to add to your swap partition. Then enlarge your swap partition.

On Tue, Dec 20, 2016, 11:51 Caetano Veyssières <caeta...@gmail.com> wrote:

Thanks. I think I will do a fresh Linux install with more swap (I need to do a fresh install anyway).
But are we sure Hugin uses swap files ? If I have an 8GB swap partition and a 12 GB file, will it work the same as having a 20 GB partition ?

--
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
---
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hugin-ptx+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hugin-ptx/f20c11c6-f63b-4511-8c22-c2983d2a9b2a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Caetano Veyssières]

So what you're saying is that 8 GB of swap partition + 12 GB of swap file (added to 8 GB of RAM that I forgot to mention at the begining) are way enough for Hugin, even if the image is huge, since the generated panorama isn't going in the memory but instead only in disk space ?
I would've thought Hugin would need to put it in memory entirely at first in order to process it.

I wanted to set a swap partition with Gparted but a warning told me that it would erase all the data in the selected partition. So I guess I have no choice but to do it on my data partiton if I don't want to do a fresh install, right ? But since my data partition is also on a different hard drive, could the new swap partition still merge with the first one ?

Temporary files, such as ones created in /tmp, are NOT memory.

Not sure what files you're refering to in this context. Do you mean files Hugin generates as it's processing the image and deletes at the end ?

[David W. Jones]

On 12/20/2016 04:45 PM, Caetano Veyssières wrote:
> So what you're saying is that 8 GB of swap partition + 12 GB of swap

> file (added to 8 GB of RAM that I forgot to mention at the beginning) are

> way enough for Hugin, even if the image is huge, since the generated
> panorama isn't going in the memory but instead only in disk space ?
> I would've thought Hugin would need to put it in memory entirely at
> first in order to process it.

That's the thing about swap space. As far as a typical program running
on the system is concerned, swap space IS in memory. If memory is full
when Hugin asks the system for more memory, the system pauses Hugin,
picks the least-used (or least-recently-used - I don't know anything
real about memory allocation/swapping) page of real RAM, writes what's
currently in it into swap on the disk, then gives the now-available page
of RAM to Hugin.

That swap process can happen again and again, as long as Hugin or other
programs are asking for more memory than the 8GB RAM that you have (and
the system doesn't run out of swap space). Should Hugin need a piece of
memory back from swap, it just tries to access the memory. The system
pauses Hugin, swaps some other area of memory out, copies the old memory
back into real RAM, and unpauses Hugin.

Now disks, even on an SSD, are way slower than RAM. So the result is
that the processing gets done a lot slower. But if time isn't of the
essence, theoretically a system with little RAM could work with enormous
images if it had enough swap space available.

On my old 32-bit 2GB RAM laptop, I used to run cpfind from command line
on 6MPix RAW images at full resolution. Processing a single image used
1.9GB of RAM. Took awhile, but it got done.

I even stitched a panorama that ended up being a 750MB 48-bit TIFF file
on that system. Took the system something like a day, if I recall
correctly. Been awhile!

You refer to a swap file. I've never used that setup. I've always set up
my systems with a separate swap partition, sized to match the amount of
RAM in my system, or double the RAM. According to an article I found on
Linux.com, a swapfile is just like a swap partition. They're both
treated the same way.

> I wanted to set a swap partition with Gparted but a warning told me that
> it would erase all the data in the selected partition. So I guess I have
> no choice but to do it on my data partiton if I don't want to do a fresh
> install, right ? But since my data partition is also on a different hard
> drive, could the new swap partition still merge with the first one ?

Presumably your swap file is either on your system partition, or your
data partition. If you want more swap space in it, I guess you can
increase it's size. (Sorry, I don't know how to do that.) You can ignore
what I said about creating a swap partition. You don't need it.

> Temporary files, such as ones created in /tmp, are NOT memory.
>
> Not sure what files you're refering to in this context. Do you mean
> files Hugin generates as it's processing the image and deletes at the end ?

I was just using it as an example. Only time I've noticed Hugin making
any files in /tmp is when it's running commands like optimize or such.
Then it creates a /tmp version of a PTO file and runs the appropriate
command on it. Or something like that. People who know the insides of
Hugin and how it interacts with commands like cpfind, etc, would be
better resources.

When Hugin is remapping images during the stitching process, it produces
the remapped images temporarily in the same directory the PTO file is
in. Those files use disk space as ordinary files. Hugin reads and write
them as ordinary files, then deletes them when it's done. So I suppose
it's possible for Hugin to run out of space on that disk if it has to
write a lot of remapped images.

If I recall correctly, though, didn't someone else post something about
enblend only working with image sizes 2GB in size or less? (Or was that
a different thread?) If so, no amount of memory or swap will help,
because the program itself can't handle an image larger than that.
Although that always bugs me. All the systems currently in our house run
64-bit Linux - even my wife's old Atom-powered netbook. I'd expect a
64-bit executable to support 64-bit addressing when it comes to file
sizes, too. But I don't know, enblend is doing powerful stuff under the
hood, so maybe there are other reasons for the 2GB image size limit.

--
David W. Jones
gnome...@gmail.com
wandering the landscape of god
http://dancingtreefrog.com

[cspiel]

David,

    Thanks for the extensive explanation of core memory vs. swap
partition vs. swap file.

On Wednesday, 21 December 2016 08:44:56 UTC+1, GnomeNomad wrote:
--- snip ---

But I don't know, enblend is doing powerful stuff under the
hood, so maybe there are other reasons for the 2GB image size limit.

    It is the other way around.  ;)  Enblend and Enfuse use `int' weaklings
instead of `std::ptrdiff' in way too many corners of the code to index into an
image .  Thus we even have the 2^(32 - 1) pixel limit on 64-bit boxes.  Also
see
    http://enblend.sourceforge.net/enblend.doc/enblend_4.2.xhtml/enblend.html#sec%3Aknown-limitations


Cheers,
    Chris

[Caetano Veyssières]

I tried with a 1.8 Gigapixel output image and this time the builtin blender didn't fail right at the start but failed after some time.
Seems there's also a swap issue (I hope so, otherwise I'd have no idea what to do).

[David W. Jones]

Ah, yes, I think you posted that earlier. Can't that be changed fairly
simply? But maybe that would run into similar problems in 3rd party
libraries, too.

[David W. Jones]

I guess you just increase the size of your swap file and try again?

[Caetano Veyssières]

Just moved from Linux mint to OpenSuse with 30 GB of Swap partition and the 1.8 Gigapixel image got done ! (the .tif file is 3.3 GB). The cool thing is Hugin was already installed. Seems like OpenSuse KDE has it by default. Maybe this means it's the best distro for this program. I'm currently hesitating between OpenSuse and Manjaro and just added a new topic to know which Linux distro is the most adapted here.

[Gnome Nomad]

Excellent! :)

On Fri, Dec 23, 2016, 04:09 Caetano Veyssières <caeta...@gmail.com> wrote:

Just moved from Linux mint to OpenSuse with 30 GB of Swap partition and the 1.8 Gigapixel image got done ! (the .tif file is 3.3 GB). The cool thing is Hugin was already installed. Seems like OpenSuse KDE has it by default. Maybe this means it's the best distro for this program. I'm currently hesitating between OpenSuse and Manjaro and just added a new topic to know which Linux distro is the most adapted here.

--

A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
---
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hugin-ptx+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/hugin-ptx/43e06fe9-add5-43b0-8d4b-0fd93feef484%40googlegroups.com.