Hugin 14.0.0 optimize never completes

[Bob Mahar]

This is Hugin 14.0.0 on 32-bit and 64-bit linux platforms.  The 64-bit one is 8 fast cores, 12GB of ram, server class machine with a 20 drive disk array.

I have a couple projects I'm trying to stitch for which the Optimize ( Position + Barrel Distortion ) from the Hugin GUI never completes.    It gets slower, and slower, and finally subsequent itterations are hours apart.   I see no indication of the Hugin process running out of memory, or swapping to disk, it just seems to have gone into the weeds.

I'm not new to Hugin and have used it since its inception, so I have a general sense for what "normal" is.   In some instrances, projects which work and those which fail are very similar - same general venue, similar shooting technique and lens used, etc.   So I expect its just balled up some how.

So the key questions: 

1. How do you troubleshoot this?   Debug switches, setting?  Its obviously Hugin is doing a LOT of something.   The CPU utilization shows up in the hugin process itself.
2. The Hugin UI appears to be single threaded.   So the whole thing locks up when it is deep into a optimization run.   This is a not good design pattern for a UI, as any failure of the Optimize risks user data   The user's only option is to kill -9 the hugin GUI.   I'm not sure how horrible that would be to accomplish.  
3.  I suspect the issue is in the barrel distortion optiomation.   If you optimize for position only, it does complete.   But barrel distortion corrections are needed for the pano to stitch properly.  Is there a workflow which allows you to front end the barrel distortion analysis / correction so that this form of optimization is not required?
4.  Any other causes for this which can be avoided?   "Too many" control points?   Bad versions of support libraries?

 Thanks!

-- Bob

[Terry Duell]

Hello Bob,

On Mon, 02 Mar 2015 02:10:18 +1100, Bob Mahar <muhl...@gmail.com> wrote:

> This is Hugin 14.0.0 on 32-bit and 64-bit linux platforms. The 64-bit
> one is 8 fast cores, 12GB of ram, server class machine with a 20 drive
> disk
> array.
>
> I have a couple projects I'm trying to stitch for which the Optimize (
> Position + Barrel Distortion ) from the Hugin GUI never completes. It
> gets slower, and slower, and finally subsequent itterations are hours
> apart. I see no indication of the Hugin process running out of memory,
> or swapping to disk, it just seems to have gone into the weeds.
>

I would assume that these are big projects, as I have never run into
similar behaviour with small to moderately sized projects.
Can you attach the .pto file for the project, saved prior to optimisation
for positions and barrel?

Cheers,
--
Regards,
Terry Duell

[Bob Mahar]

On Sunday, March 1, 2015 at 4:47:47 PM UTC-5, Tduell wrote:

Hello Bob,

I would assume that these are big projects, as I have never run into  
similar behaviour with small to moderately sized projects.
Can you attach the .pto file for the project, saved prior to optimisation  
for positions and barrel?

Big is relative, its ~40 exposure stacks of 5 shots each.   I've had projects with several times more images, and on much less competent hardware.   The pto is attached.  

Thanks for taking a look!

-- Bob

[Terry Duell]

Hello Bob,

I'm running a basic optimisation now (positions), which is grinding along
exceedingly slowly.
In the meantime I have run your .pto through checkpto, the result is
attached.
Note that there are two unconnected image groups, each group shown in [ ].
You might want to check that this is correct. I suspect unconnected image
groups may cause problems, and think I have come across them before, but
can't for the life of me recall what project or where to be able to dredge
up any more useful info.
I can't pursue that any further as I don't have the images to eyeball.
Could you check that out and let's know how you get on?

[T. Modes]

Am Montag, 2. März 2015 04:29:46 UTC+1 schrieb Tduell:

Note that there are two unconnected image groups, each group shown in [ ].
You might want to check that this is correct. I suspect unconnected image  
groups may cause problems, and think I have come across them before, but  
can't for the life of me recall what project or where to be able to dredge  
up any more useful info.

Beside the unconnected image there is another issue with this project: the images in each stack are connected by control points. But at the same time the position of the images in each stack is linked. Please use only one of this:
* Either use control points between the images of each stack. Then allow the position of each image in the stack to be optimized (do not link them).
* Or link the position of the images in the stack. In this case don't assign control points between the individual images of each stack.
But not both at the same time. This is probably too much for the optimizer.

[Bob Mahar]

I learn something new every day!   checkpto is my new friend

I rarely use the quick preview or the wizard thing any more which shows this.   Yes, that will certainly mess up things.   I'm sure that is it.  I'll try and clean it up tonight and see what happens.  But I'm sure you are correct.   The unattached image is a piece of blue sky.

On another problem child project, I am noticing that the images are being rotated oddly.  The Pentax K-5 I use does indicate orientation, and when pointing straight down ( the first course for a stereographic projection ) the orientation may be stupidly assigned by the camera.   Is there a means to force the images to be displayed the same way in the control point editor?   Or to ignore the EXIF orientation hints?

Thanks!

[tbransco]

This rotation likely occurred after you optimized, correct?  I've seen it recommended here that before you optimize, it's always a good idea to add at least one vertical line to one of your images using the control point editor.  Personally, my panos don't seem to ever have vertical lines in them with which I can do this, but it might work for you.  After you've optimized, I'm not sure what you can do, but there's far bigger brains on here than mine!

Cheers, and good luck!
Terry B

[Terry Duell]

On Tue, 03 Mar 2015 04:26:04 +1100, T. Modes <Thomas...@gmx.de> wrote:


>
> Beside the unconnected image there is another issue with this project:
> the images in each stack are connected by control points. But at the
> same time the position of the images in each stack is linked. Please use
> only one of this:
> * Either use control points between the images of each stack. Then allow
> the position of each image in the stack to be optimized (do not link
> them).
> * Or link the position of the images in the stack. In this case don't
> assign control points between the individual images of each stack.
> But not both at the same time. This is probably too much for the
> optimizer.
>

Having pondered a bit on this project, I came to a similar conclusion.
I would reckon that a good approach for a project of this size (number and
depth of stacks), would be to pre-align each stack (align_image_stack),
which with a bit of shell scripting magic could make use a bit of parallel
processing, then align and stitch the stacks linked by position.
This is something which might be better done within hugin, and would be
good GSOC project if we can ever get back into that.

[Terry Duell]

Bob, If your still keeping up with this, see below,

On Tue, 03 Mar 2015 09:01:55 +1100, Terry Duell <tdu...@iinet.net.au>
wrote:

[snip]

>
> Having pondered a bit on this project, I came to a similar conclusion.
> I would reckon that a good approach for a project of this size (number
> and depth of stacks), would be to pre-align each stack
> (align_image_stack), which with a bit of shell scripting magic could
> make use a bit of parallel processing, then align and stitch the stacks
> linked by position.
> This is something which might be better done within hugin, and would be
> good GSOC project if we can ever get back into that.
>

The question of automatically pre-aligning intrigued me sufficiently to
see if I could cobble up some scripts to do the pre-aligning of the stacks.
The attached scripts are written for unix sh or linux bash and also
require awk.
Put both scripts in the dir with your project stacks, but only have those
images in that dir.
From a terminal window opened in the image dir run...
"./ais-script2.sh myprefix" where myprefix is the output prefix you want
for the aligned image stacks.
I'm sure the whole business could be done much better than the way these
scripts do it, but at the moment I am not really concerned with that level
of detail or correctness, so bear that in mind if you want to comment on
how it could be done better :-)
No attempt has been made at this stage to use multiple threads.
I don't have any suitable projects that I can test this on, but it looks
like it should work.
Bob, if you are game, could you give it a try with your project (the one
that has been the subject of this thread), and let me know how you get on.
You will need to make some minor edits to ais-script2.sh to actually make
the calls to align_image_stack.

[Terry Duell]

On Tue, 03 Mar 2015 14:43:09 +1100, Terry Duell <tdu...@iinet.net.au>

wrote:

> Bob, If your still keeping up with this, see below,
>
> On Tue, 03 Mar 2015 09:01:55 +1100, Terry Duell <tdu...@iinet.net.au>

[snip]

> The question of automatically pre-aligning intrigued me sufficiently to
> see if I could cobble up some scripts to do the pre-aligning of the
> stacks.
> The attached scripts are written for unix sh or linux bash and also
> require awk.

Of course as soon as I'd posted these scripts it became obvious that each
stack would be overwritten by the next output from align_image_stack.
A minor modification has been made to the ais-script2.sh to include a
counter value in the prefix.
It looks like it works OK here...but then the last one did too :-)

[Terry Duell]

On Tue, 03 Mar 2015 15:35:06 +1100, Terry Duell <tdu...@iinet.net.au>
wrote:

[snip]

> It looks like it works OK here...but then the last one did too :-)
>

Hah! Famous last words.
It still wasn't right.
I have now run a real-live test with a project, 4 stacks, 3 deep, and the
attached script worked here.
Note this script is configured for stacks 3 deep.
It now runs the alignment of each stack in the background, so will use all
cores available.
I think I'm done with this.

[T. Modes]

Am Dienstag, 3. März 2015 06:53:53 UTC+1 schrieb Tduell:

Note this script is configured for stacks 3 deep.
It now runs the alignment of each stack in the background, so will use all  
cores available.

 Why? Align_image_stack use already all available cores. Running several instances of align_image_stacks parallel does not help, it can even slow down the whole process.

Thomas

[Bob Mahar]

Removing the orphaned groups or adding control points to connect them fixed the problem, and I have both rendering now, they will look great.   So it was entirely the disconnected groups which were causing the issue in both projects.   Thanks!     I tend to leave the control points in exposure brackets as it will provide more control points in blown out or shadow areas.  And generally, I've not had issues with optimization taking too long.  I guess the change is in the way I do things.  

Using the quick preview, its very obvious when you have multiple image groups.  But I rarely use that any more.  I just need to pay more attention to this.

-- Bob

[Terry Duell]

Hello Thomas,

Interesting. I don't think that's what I'm seeing here.
When running align_image-stack (ais) via the shell script for 4 stacks,
each ais run must complete before the next can start. When run this way I
only see one core of my system running at 100%.
By running each instance of ais as a background task, the script will
start the next instance of ais immediately after starting the previous,
and I'm seeing all cores running 100%.
I haven't run a timer. I'll see if I can do that.

[Terry Duell]

On Wed, 04 Mar 2015 07:53:36 +1100, Terry Duell <tdu...@iinet.net.au>
wrote:

[snip]

> I haven't run a timer. I'll see if I can do that.

OK, just run my script on my test project, and the result of running the
script via the 'time' command, are as follows;

each run of ais run as background task.

[terry@localhost mytest]$ time ./ais-script3.sh mytest
align_image_stack -a mytest1 IMGP2041.JPG IMGP2042.JPG IMGP2043.JPG
align_image_stack -a mytest2 IMGP2044.JPG IMGP2045.JPG IMGP2046.JPG
align_image_stack -a mytest3 IMGP2047.JPG IMGP2048.JPG IMGP2049.JPG
align_image_stack -a mytest4 IMGP2050.JPG IMGP2051.JPG IMGP2052.JPG
Written aligned images to files with prefix "mytest3"
Written aligned images to files with prefix "mytest2"
Written aligned images to files with prefix "mytest1"
Written aligned images to files with prefix "mytest4"
done

real 0m28.325s
user 3m31.727s
sys 0m2.357s

each run if ais as sequential task.

[terry@localhost mytest]$ time ./ais-script3.sh mytest
Written aligned images to files with prefix "mytest1"
align_image_stack -a mytest1 IMGP2041.JPG IMGP2042.JPG IMGP2043.JPG
Written aligned images to files with prefix "mytest2"
align_image_stack -a mytest2 IMGP2044.JPG IMGP2045.JPG IMGP2046.JPG
Written aligned images to files with prefix "mytest3"
align_image_stack -a mytest3 IMGP2047.JPG IMGP2048.JPG IMGP2049.JPG
Written aligned images to files with prefix "mytest4"
align_image_stack -a mytest4 IMGP2050.JPG IMGP2051.JPG IMGP2052.JPG
done

real 0m40.361s
user 3m26.731s
sys 0m2.041s

'real' time is the 'wall clock' time, so running the ais tasks as
background in the shell script is quite a bit quicker.

[T. Modes]

Am Dienstag, 3. März 2015 21:53:51 UTC+1 schrieb Tduell:

Interesting. I don't think that's what I'm seeing here.
When running align_image-stack (ais) via the shell script for 4 stacks,  
each ais run must complete before the next can start. When run this way I  
only see one core of my system running at 100%.

Strange. Theres is something with your build, or a bug in tue build system, or you have set the environment variable OMP_NUM_THREADS to 1. Align_image_stacks runs on all cores here.

Thomas

[Terry Duell]

Hello Thomas,

OK, thanks for that info.
I'll do some checking.

[Terry Duell]

On Wed, 04 Mar 2015 08:55:33 +1100, Terry Duell <tdu...@iinet.net.au>
wrote:

I haven't found anything obviously wrong with the build process, so tried
running my script again and monitored cpu activity with a different tool,
that gives a much better view of what's going on.
align_image_stack does indeed use all the cores for quite short periods of
time, with periods in between where activity drops back to one core. When
I run the script with calls to ais run as background tasks the cpu
activity peaks at 100% (all cores) for most of the time.
I note that the output images from ais don't have any the exif data that
hugin uses to automatically detect stacks.

[T. Modes]

Hi Terry,

Am Mittwoch, 4. März 2015 02:22:25 UTC+1 schrieb Tduell:

align_image_stack does indeed use all the cores for quite short periods of  
time, with periods in between where activity drops back to one core. When  
I run the script with calls to ais run as background tasks the cpu  
activity peaks at 100% (all cores) for most of the time.

thanks for checking.
This matches the code: align_image_stacks does all calculation in multi-threading context. Only the image loading is single threaded.
This should be only short times. So I assume the used resource monitor has some problem to display the core usage in real-time.

Also your timing speak that align_image_stack is using all cores:
Sequential execution: 40 s
Parallel execution: 30 s

This is an improvement of "only" 25 %. So 75 % of the time align_image_stack is already running at full throttle.

Or an other calculation: in sequential execution each stack takes 10 s. But when running the processes parallel it takes 30 s to calculate the same stack.

The massive parallel execution of several align_image_stack can so even slower than the sequential execution when bigger and/or more images are used. Then the memory usage become the bottleneck.

Thomas