enblend performance / multiprocess safety
125 views
Skip to first unread message
Jani
Jun 28, 2020, 4:35:05 PM6/28/20
to hugin and other free panoramic software
Background:
I'm generating (on Ubuntu 20.04 LTS & Hugin 2019.2.0.b690aa0334b5) a sequence of 12947x2483 OpenEXR 360-panoramas from 4x OpenEXR image streams of 6144x3456 each.
The good parts (nona):
Nona works well! I can fork 12 processes and speed up the "nona -r hdr -m EXR_m -o out rig1.pto A.exr B.exr C.exr D.exr" part from 56s per frame to 7s per frame (on average).
The problem (enblend):
To get the best final results, I need to use enblend.
I pre-saved the masks hoping it would speed up the process a bit so using enblend like this:
enblend --load-masks=rig1/mask-%n.tif -v -f12947x2483 --output=pano0001.exr pano0001_stack_hdr_0000.exr pano0001_stack_hdr_0001.exr pano0001_stack_hdr_0002.exr pano0001_stack_hdr_0003.exr
Problem is that if I try to split up the enblend part to multiple processes, enblend fails randomly with signal(2) and signal(9). Enblend works well as long as I call it exactly from one process at a time, so I'm very sure the issue is related to calling it multiple times simultaneous. The file names are prefixed uniquely with frame number so there shouldn't be any collisions.
Right now my compromise is to split up nona part to 12 processes and then queue up enblend part and do it frame-by-frame from a single process. That is far from ideal because the enblend is the bottleneck in the first place.
Any thoughts or suggestions?
Thanks,
Jani
David W. Jones
Jun 28, 2020, 6:05:55 PM6/28/20
to hugi...@googlegroups.com
Just some thoughts.
My understanding is enblend already supports multiprocessing. Maybe the manual attempt at multiprocessing via fork is interfering with that?
What version of enblend are you using? There was a time when we had to use enblend-mp to get enblend's MP support.
Hugin is up to 2020.something now. Maybe an update is in order?
--
David W. Jones
gnome...@gmail.com
wandering the landscape of god
http://dancingtreefrog.com
Sent from my Android device with F/LOSS K-9 Mail.
My understanding is enblend already supports multiprocessing. Maybe the manual attempt at multiprocessing via fork is interfering with that?
What version of enblend are you using? There was a time when we had to use enblend-mp to get enblend's MP support.
Hugin is up to 2020.something now. Maybe an update is in order?
--
David W. Jones
gnome...@gmail.com
wandering the landscape of god
http://dancingtreefrog.com
Sent from my Android device with F/LOSS K-9 Mail.
Jani
Jun 29, 2020, 1:28:12 AM6/29/20
to hugin and other free panoramic software
Thanks, I didn't realize the 2020 version was already in beta. I will try that one. Or maybe I try to compile Hugin & tools from sources, might be good for debugging this.
Yeah I agree with you, it looks like the multiprocessing support in enblend somehow gets messed up when called from multiple processes. Kind of funny. Nona works without any issues.
Jani
T. Modes
Jun 29, 2020, 11:13:45 AM6/29/20
to hugin and other free panoramic software
Am Sonntag, 28. Juni 2020 22:35:05 UTC+2 schrieb Jani:
Nona works well! I can fork 12 processes and speed up the "nona -r hdr -m EXR_m -o out rig1.pto A.exr B.exr C.exr D.exr" part from 56s per frame to 7s per frame (on average).
That's somehow surprising. Nona is already running multi threads parallel. So running multiple instances at the same time should not bring that difference.
Jani
Jun 29, 2020, 12:19:47 PM6/29/20
to hugin and other free panoramic software
Kind of makes sense, still, since it's always more efficient to split work by bigger workloads (fully separate images) vs. divide single image to individual threads. But yeah, surprisingly big speedup.
In case you're interested, I just used Python (3.8) Thread class, and each Thread just calls subprocess.run(["nona", ...], capture_output=True), so nothing special in implementation.
David W. Jones
Jun 29, 2020, 11:33:05 PM6/29/20
to hugin-ptx
I don't know the internals of the programs. It's possible that nona is
actually only working on one image at a time while enblend is working
out a seam between two images (possibly more? depending on image
overlaps). Since it's already multiprocessing as it does this, my guess
is it uses signals to coordinate the seaming process.
When you have multiple enblends running via fork, maybe the signals are
no longer good for coordinating; maybe they cross from one enblend to
another enblend session. The second session might well be in a different
state than the first session, leading to confusion and errors.
Just guessing, I'm sure the enblend and nona developers are laughing
right now. ;) Anyway, I have an i9 - before that I had an i7 - and
enblend happily uses all cores and threads without me having to do anything.
On 6/28/20 7:28 PM, Jani wrote:
> Thanks, I didn't realize the 2020 version was already in beta. I will
> try that one. Or maybe I try to compile Hugin & tools from sources,
> might be good for debugging this.
>
> Yeah I agree with you, it looks like the multiprocessing support in
> enblend somehow gets messed up when called from multiple processes. Kind
> of funny. Nona works without any issues.
>
>
> Jani
>
> On Sunday, June 28, 2020 at 5:05:55 PM UTC-5, GnomeNomad wrote:
>
> On June 28, 2020 10:35:05 AM HST, Jani <kaj...@gmail.com
My password is the last 8 digits of π.
actually only working on one image at a time while enblend is working
out a seam between two images (possibly more? depending on image
overlaps). Since it's already multiprocessing as it does this, my guess
is it uses signals to coordinate the seaming process.
When you have multiple enblends running via fork, maybe the signals are
no longer good for coordinating; maybe they cross from one enblend to
another enblend session. The second session might well be in a different
state than the first session, leading to confusion and errors.
Just guessing, I'm sure the enblend and nona developers are laughing
right now. ;) Anyway, I have an i9 - before that I had an i7 - and
enblend happily uses all cores and threads without me having to do anything.
On 6/28/20 7:28 PM, Jani wrote:
> Thanks, I didn't realize the 2020 version was already in beta. I will
> try that one. Or maybe I try to compile Hugin & tools from sources,
> might be good for debugging this.
>
> Yeah I agree with you, it looks like the multiprocessing support in
> enblend somehow gets messed up when called from multiple processes. Kind
> of funny. Nona works without any issues.
>
>
> Jani
>
> On Sunday, June 28, 2020 at 5:05:55 PM UTC-5, GnomeNomad wrote:
>
> On June 28, 2020 10:35:05 AM HST, Jani <kaj...@gmail.com
Jani
Jun 30, 2020, 12:25:53 PM6/30/20
to hugin and other free panoramic software
As far as I can tell from Hugin execution log, enblend is never executed same time as nona. So they don't coordinate anything but enblend is (optional) post-processing step. Nona seems to have internal blending functionality as well, it's just that enblend produces higher quality output at least in my case.
David W. Jones
Jun 30, 2020, 1:25:49 PM6/30/20
to hugi...@googlegroups.com
I wasn't referring to enblend coordinating with nona. I was referring to one enblend somehow interacting (via signals) with one or more of the forked enblends.
Jani
Jun 30, 2020, 2:00:26 PM6/30/20
to hugin and other free panoramic software
Yeah it shouldn't work like that. The parent gets process id of the forked process (or thread) and can coordinate directly using those. I am guessing the issue is related to some shared resource e.g. dynamically linked library. Enblend on Windows seem to work, I have only issues with the Linux build.
David W. Jones
Jun 30, 2020, 6:10:41 PM6/30/20
to hugi...@googlegroups.com
Hmm. Maybe compile enblend with static libs? Then each forked enblend would have its own copy of the libraries? I don't know, could be quite wrong about that.
On June 30, 2020 8:00:26 AM HST, Jani <kaj...@gmail.com> wrote:
>Yeah it shouldn't work like that. The parent gets process id of the
>forked process (or thread) and can coordinate directly using those.
>I am guessing the issue is related to some shared resource e.g.
>dynamically linked library. Enblend on Windows seem to work, I
>have only issues with the Linux build.
>
>On Tuesday, June 30, 2020 at 12:25:49 PM UTC-5, GnomeNomad wrote:
>>
>> I wasn't referring to enblend coordinating with nona. I was referring
>> to one enblend somehow interacting (via signals) with one or more
>> of the forked enblends.
>>
On June 30, 2020 8:00:26 AM HST, Jani <kaj...@gmail.com> wrote:
>Yeah it shouldn't work like that. The parent gets process id of the
>forked process (or thread) and can coordinate directly using those.
>I am guessing the issue is related to some shared resource e.g.
>dynamically linked library. Enblend on Windows seem to work, I
>have only issues with the Linux build.
>
>On Tuesday, June 30, 2020 at 12:25:49 PM UTC-5, GnomeNomad wrote:
>>
>> I wasn't referring to enblend coordinating with nona. I was referring
>> to one enblend somehow interacting (via signals) with one or more
>> of the forked enblends.
>>
Jani
Jun 30, 2020, 10:16:04 PM6/30/20
to hugin and other free panoramic software
I didn't get enblend to compile, seems like some C++ 17 issues. I installed all dependencies on Ubuntu 20.04 LTS with apt install xxx and then followed the README instructions. Hugin compiled fine, though.
David W. Jones
Jun 30, 2020, 10:56:56 PM6/30/20
to hugi...@googlegroups.com
I've never tried to compile enblend.
Maybe someone else has ideas about enblend and compiling with C++ 17?
On June 30, 2020 4:16:04 PM HST, Jani <kaj...@gmail.com> wrote:
>I didn't get enblend to compile, seems like some C++ 17 issues. I
>installed
>all dependencies on Ubuntu 20.04 LTS with apt install xxx and then
>followed
>the README instructions. Hugin compiled fine, though.
>
>On Tuesday, June 30, 2020 at 5:10:41 PM UTC-5, GnomeNomad wrote:
>>
>> Hmm. Maybe compile enblend with static libs? Then each forked enblend
>
>> would have its own copy of the libraries? I don't know, could be
>quite
>> wrong about that.
>>
>> On June 30, 2020 8:00:26 AM HST, Jani <kaj...@gmail.com
><javascript:>>
Maybe someone else has ideas about enblend and compiling with C++ 17?
On June 30, 2020 4:16:04 PM HST, Jani <kaj...@gmail.com> wrote:
>I didn't get enblend to compile, seems like some C++ 17 issues. I
>installed
>all dependencies on Ubuntu 20.04 LTS with apt install xxx and then
>followed
>the README instructions. Hugin compiled fine, though.
>
>On Tuesday, June 30, 2020 at 5:10:41 PM UTC-5, GnomeNomad wrote:
>>
>> Hmm. Maybe compile enblend with static libs? Then each forked enblend
>
>> would have its own copy of the libraries? I don't know, could be
>quite
>> wrong about that.
>>
>> On June 30, 2020 8:00:26 AM HST, Jani <kaj...@gmail.com
Terry Duell
Jul 1, 2020, 12:26:44 AM7/1/20
to hugi...@googlegroups.com
On Tue, 2020-06-30 at 16:56 -1000, David W. Jones wrote:
> I've never tried to compile enblend.
>
> Maybe someone else has ideas about enblend and compiling with C++ 17?
>
I just tried to build enblend-4.3.0-1542 on Fedora 32, and it failed early on
> I've never tried to compile enblend.
>
> Maybe someone else has ideas about enblend and compiling with C++ 17?
>
reporting that it had found exiv2-0.12 which wasn't sufficient.
I then attempted to install exiv2-0.27, and my system tells me it is already
installed, so it looks like the initial tests are going awry.
Beyond my ken, so hopefully another someone can help?
Cheers,
--
Terry Duell <tdu...@iinet.net.au>
Terry Duell
Jul 1, 2020, 12:35:43 AM7/1/20
to hugi...@googlegroups.com
On Wed, 2020-07-01 at 14:26 +1000, Terry Duell wrote:
> On Tue, 2020-06-30 at 16:56 -1000, David W. Jones wrote:
>
> > I've never tried to compile enblend.
> >
> > Maybe someone else has ideas about enblend and compiling with C++ 17?
> >
>
> I just tried to build enblend-4.3.0-1542 on Fedora 32, and it failed early on
> reporting that it had found exiv2-0.12 which wasn't sufficient.
Whoops, my mistake...the report was said...
> On Tue, 2020-06-30 at 16:56 -1000, David W. Jones wrote:
>
> > I've never tried to compile enblend.
> >
> > Maybe someone else has ideas about enblend and compiling with C++ 17?
> >
>
> I just tried to build enblend-4.3.0-1542 on Fedora 32, and it failed early on
> reporting that it had found exiv2-0.12 which wasn't sufficient.
-- Found Exiv2 release >= 0.12
-- Found Exiv2: /usr/lib64/libexiv2.so
-- Performing Test EXIV2_COMPILES_FLAG
-- Performing Test EXIV2_COMPILES_FLAG - Failed
CMake Error at CMakeModules/FindEXIV2.cmake:145 (MESSAGE):
Found exiv2 library, but this library is not recent enough to compile
so it wants Exiv2 > 0.27, which Fedora 32 hasn't provided yet.
I seem to recall an issue like this a while ago.
David W. Jones
Jul 1, 2020, 1:03:19 AM7/1/20
to hugin-ptx
I have Debian Testing here, latest exiv2 it reports is 0.25-4. With
enblend 4.2-4. Maybe try the enblend 4.2 release instead of 4.3?
Terry Duell
Jul 1, 2020, 2:19:41 AM7/1/20
to hugi...@googlegroups.com
On Tue, 2020-06-30 at 19:03 -1000, David W. Jones wrote:
>
> Anyone tried on recent Ubuntu or Debian Testing?
>
> I have Debian Testing here, latest exiv2 it reports is 0.25-4. With
> enblend 4.2-4. Maybe try the enblend 4.2 release instead of 4.3?
>
I have just built the development version of exiv2, which git announces as 0.27-
>
> Anyone tried on recent Ubuntu or Debian Testing?
>
> I have Debian Testing here, latest exiv2 it reports is 0.25-4. With
> enblend 4.2-4. Maybe try the enblend 4.2 release instead of 4.3?
>
RC3-606-g1ea63ccb and installed that on Fedora 32, then built enblend-4.3, and
get the following messages...
In file included from /usr/include/vigra/stdconvolution.hxx:43,
from /usr/include/vigra/convolution.hxx:41,
from /home/terry/rpmbuild/BUILD/enblend-4.3.0-1542hg-
Source/src/openmp_vigra.h:32,
from /home/terry/rpmbuild/BUILD/enblend-4.3.0-1542hg-
Source/src/enfuse.h:47,
from /home/terry/rpmbuild/BUILD/enblend-4.3.0-1542hg-
Source/src/enfuse.cc:187:
/usr/include/vigra/separableconvolution.hxx:1413:13: error: ISO C++17 does not
allow dynamic exception specifications
1413 | throw(PreconditionViolation)
| ^~~~~
In file included from /usr/include/vigra/convolution.hxx:41,
from /home/terry/rpmbuild/BUILD/enblend-4.3.0-1542hg-
Source/src/openmp_vigra.h:32,
from /home/terry/rpmbuild/BUILD/enblend-4.3.0-1542hg-
Source/src/enfuse.h:47,
from /home/terry/rpmbuild/BUILD/enblend-4.3.0-1542hg-
Source/src/enfuse.cc:187:
/usr/include/vigra/stdconvolution.hxx:796:13: error: ISO C++17 does not allow
dynamic exception specifications
796 | throw(PreconditionViolation)
| ^~~~~
so hopefully that will point someone who understands this to what is causing the
C++17 errors in this version and whether they need fixing at this stage.
Wolfgang Goetz
Jul 1, 2020, 3:00:02 AM7/1/20
to hugi...@googlegroups.com
On Tue, 30 Jun 2020 19:16:04 -0700 (PDT)
Jani <kaj...@gmail.com> wrote:
> I didn't get enblend to compile, seems like some C++ 17 issues. I installed
> all dependencies on Ubuntu 20.04 LTS with apt install xxx and then followed
> the README instructions. Hugin compiled fine, though.
use vigra from git!
Jani <kaj...@gmail.com> wrote:
> I didn't get enblend to compile, seems like some C++ 17 issues. I installed
> all dependencies on Ubuntu 20.04 LTS with apt install xxx and then followed
> the README instructions. Hugin compiled fine, though.
The last official release is from 2017.
http://ukoethe.github.io/vigra/
....
git clone https://github.com/ukoethe/vigra.git
cd vigra
mkdir build
cd build
cmake ..
make
sudo make install
sudo ldconfig
....
cd enblend-code
mkdir build
cd build
cmake ..
make
[ 97%] Linking CXX executable ../bin/enblend
[ 97%] Built target enblend
[100%] Linking CXX executable ../bin/enfuse
[100%] Built target enfuse
greets
Wolfgang
>
> On Tuesday, June 30, 2020 at 5:10:41 PM UTC-5, GnomeNomad wrote:
> >
> > Hmm. Maybe compile enblend with static libs? Then each forked enblend
> > would have its own copy of the libraries? I don't know, could be quite
> > wrong about that.
> >
> > wandering the landscape of god
> > http://dancingtreefrog.com
> >
> > Sent from my Android device with F/LOSS K-9 Mail.
> >
>
> --
> > http://dancingtreefrog.com
> >
> > Sent from my Android device with F/LOSS K-9 Mail.
> >
>
> A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
> ---
> You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to hugin-ptx+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/hugin-ptx/32be713c-f182-4fdb-81e6-d297f4846bd9o%40googlegroups.com.
Jani
Jul 1, 2020, 5:40:50 PM7/1/20
to hugin and other free panoramic software
Thanks Wolfgang, that worked! One minor thing in case someone else following this thread: You need to pass cmake .. -DWITH_OPENEXR=1 when building Vigra since Hugin requires OpenEXR
> To unsubscribe from this group and stop receiving emails from it, send an email to hugi...@googlegroups.com.
Wolfgang Goetz
Jul 2, 2020, 10:21:26 AM7/2/20
to hugi...@googlegroups.com
On Wed, 1 Jul 2020 14:40:50 -0700 (PDT)
Jani <kaj...@gmail.com> wrote:
> Thanks Wolfgang, that worked! One minor thing in case someone else
> following this thread: You need to pass cmake .. -DWITH_OPENEXR=1 when
> building Vigra since Hugin requires OpenEXR
Thanks, recompiled hugin+enblend!
Jani <kaj...@gmail.com> wrote:
> Thanks Wolfgang, that worked! One minor thing in case someone else
> following this thread: You need to pass cmake .. -DWITH_OPENEXR=1 when
> building Vigra since Hugin requires OpenEXR
would have been stumbled into this next hugin update. ;-)
kr
Wolfgang
> > > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/hugin-ptx/32be713c-f182-4fdb-81e6-d297f4846bd9o%40googlegroups.com.
> >
> >
> >
>
> > https://groups.google.com/d/msgid/hugin-ptx/32be713c-f182-4fdb-81e6-d297f4846bd9o%40googlegroups.com.
> >
> >
> >
>
> --
> A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
> ---
> You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to hugin-ptx+...@googlegroups.com.
> A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
> ---
> You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
> To view this discussion on the web visit https://groups.google.com/d/msgid/hugin-ptx/38538a28-e0bd-45a1-9520-146a2b7f3418o%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages