improving Hugin's responsiveness - patch attached
Yuval Levy
One thing that bothers me about Hugin is how slow it is in opening the
fast (pun intended) preview. Start Hugin. Load a project. Click on the
fast preview icon and wait. Wait. Starr at the status bar with the (not
yet localized ;-) ) "Loading image:" messages, while one image after the
other is loaded. And the Ubuntu equivalent of the sandglass trying to
hypnotize me...
No more, I said to myself. A little bit of research later and here comes
a patch for a more proactive cacheing: rather than loading the images
(and wait for them) at the time of (first) use, I do that during the
idle loop.
Of course there is no magic: the effective loading time stays the same.
It is just the perceived responsiveness that improves - the readiness of
the application to serve me.
I tested this on my Dynobook (1.6GHz 2GB) and found it an improvement.
But I have not tested it with projects that do not fit in the cache; I
have not worked intensively with it; and I still understand too little
of Hugin's inner working.
So here is a patch implementing this, against trunk. Please test it,
especially in Windows and MacOSX; and under difficult conditions (such
as adding and removing images, and stressing the cache to its limits).
If it passes the integration tests and there is no negative feedback, I
would like to add this as-is to trunk. And maybe to 2009.4.0_RC1.
If there are negative indications, e.g. on machines with little memory,
or on very large projects that don't fit in the cache, I will add a
preference settings to turn this mode on (and it will be off by default).
The patch is attached. Currently it does not care to reload an image if
it was flushed from the cache. I am not sure if implementing this would
be a good idea, since it would run an endless loop in and out of the
cache in case the memory is not enough to cache all images.
What do *you* think?
Yuv
Kornel Benko
...
> I tested this on my Dynobook (1.6GHz 2GB) and found it an improvement.
> But I have not tested it with projects that do not fit in the cache; I
> have not worked intensively with it; and I still understand too little
> of Hugin's inner working.
>
> So here is a patch implementing this, against trunk. Please test it,
> especially in Windows and MacOSX; and under difficult conditions (such
> as adding and removing images, and stressing the cache to its limits).
>
> If it passes the integration tests and there is no negative feedback, I
> would like to add this as-is to trunk. And maybe to 2009.4.0_RC1.
>
> If there are negative indications, e.g. on machines with little memory,
> or on very large projects that don't fit in the cache, I will add a
> preference settings to turn this mode on (and it will be off by default).
The patch applies cleanly.
I tested it with some projects on ubuntu. And it is an improvement.
The immidiate response is a relief.
Sorry, I don't have MacOSX or Windows.
Kornel
--
Kornel Benko
Kornel...@berlin.de
T. Modes
> So here is a patch implementing this, against trunk. Please test it,
> especially in Windows and MacOSX; and under difficult conditions (such
> as adding and removing images, and stressing the cache to its limits).
>
preview windows shows up faster.
But I did not more tests.
Thomas
Yuval Levy
T. Modes wrote:
> It compiles in windows. At a first test no crash occurs and the
> preview windows shows up faster.
> But I did not more tests.
I tried with a project that is too big to fit in the ImageCache and I
did not find it to be any slower than without the patch. The bottlenecks
could / would be loading the image from disk, and since my dynobook has
an old 5400RPM notebook drive I would say that it is likely that there
is no significant penalty in applying the patch. But we need more tests,
especially on low-end machines (to determine bottlenecks) and on
high-end machines with big projects (to determine if the usefulness scales).
Yuv
Harry van der Wolf
Well done.
It compiles and works fine as well on OSX, both via a cmake build and an XCode build.
It seems a bit faster but I don't have big projects at hand.
Harry
James Legg
> Hi all,
>
> One thing that bothers me about Hugin is how slow it is in opening the
> fast (pun intended) preview. Start Hugin. Load a project. Click on the
> fast preview icon and wait. Wait. Starr at the status bar with the (not
> yet localized ;-) ) "Loading image:" messages, while one image after the
> other is loaded. And the Ubuntu equivalent of the sandglass trying to
> hypnotize me...
>
> No more, I said to myself. A little bit of research later and here comes
> a patch for a more proactive cacheing: rather than loading the images
> (and wait for them) at the time of (first) use, I do that during the
> idle loop.
This doesn't help me for large projects.
Big images take a long time to load. Since you do this in the idle loop,
the UI actually gets less responsive. For example: I load a project and
click on the optimiser tab. Hugin is currently loading an image, so I
have to wait for it to finish the image it is on before it can work on
drawing the optimiser tab. Before, it could show the optimiser tab
straight away. This extra delay might be fine for small images, but it
takes a while to decode the huge jpegs many cameras produce.
Ideally, one would solve this problem by loading the images in another
thread with a lower priority.
Another issue I see is that it eats memory. If I load a project where
all the decoded images don't fit into RAM, this makes my whole system
unresponsive as it eventually it starts moving stuff I'm trying to use
to swap memory.
If I try to load the fast preview while this is going on, the following
happens:
* The preview gets the first image from the image cache. It is now
loaded, where it wasn't before, saving a file load.
* The preview tries to free up some space, in case there are more
images in the cache than fit the allowed memory. There is, so
the image cache frees most of the images that have been
preloaded.
* The preview tries to get the second image from the image cache.
There is a high probability it was freed by the last step, so it
loads it off the disk again.
* The preview again asks the image if it would like to free some
memory. It is likely to free the image accessed the earliest,
which is another preloaded one.
* ...
So it only saves one file load if left too long to preload images.
Although I think the first image is loaded to show in the control points
tab when the project is loaded, so if I load a project, wait for the
cache to fill up, and then press the fast preview button, not even that
has made it appear faster than it used to.
After a while, the idle task has loaded all the images. Most of them
have since been freed by something else I used in the GUI, so many
images aren't loaded now.
The image tab's previews and the old preview use the small image cache,
which requires scaling the large images. So there is still a pause that
could be removed by doing something in the background.
I have some suggestions to get around these issues:
1. Load images in a low priority thread separate to the GUI thread
if possible.
2. Load the small images from the small image cache. These are not
freed, as they occupy a small amount of memory each. They are
used in the images tab and the traditional preview.
3. Only load the small images in reverse order. The small images
are generated from the large ones, so the large images will be
loaded. However when space is getting tight, the larger ones
will be freed. When you have finished you'll have all the small
ones, and just the lower numbered large images. These are
generally used first by algorithms that require all the images,
so if the user waits long enough before starting one of these
algorithms they will run quicker.
I wrote the fast preview to use the large image cache to create the
textures it uses. This is so that if you are, for instance, just
remapping an equirectangular pano to a little planet, the image is
sharper as it knows it can allocate more texture memory to it.
We could make the fast preview use the small image cache for the smaller
textures so that the above routine has a better effect on performance of
displaying the fast preview: All the textures could be derived from the
small image cache on a sufficiently large project.
Textures have to have power of two sizes, and the small image cache
scales images down by a half continuously until they are less than or
equal to 512 pixels in each direction. So for instance a 4000 by 3000
image gets scaled to 500 by 375, and the nearest texture size that
doesn't interpolate between pixels is 256 by 256, so this is the largest
texture that can be used from the small image cache for that image. The
amount of texture memory the fast preview aims for means that for a
project with equal size images, you would need at least 32 images like
this for it to use 256 by 256 or smaller textures for each of them. Any
projects with less than 32 images will (unless the images are different
sized) therefore still need the larger images to get the level of image
detail we currently have in the fast preview. It is worse if the images
are multiples of standard screen sizes, because they are often scaled
down to 320 by 240, which means the texture has to be 256 by 128 or
less, and you'll need 64 images to reach the point where the small image
cache is all you need.
I guess this means a small project will have all the large images in the
cache and therefore load quickly, a medium size project will have to
load some of the images twice, and a large project will only need the
small image cache and therefore load quickly. This seems better to me.
I'll try to make a patch to use the small image cache in the fast
preview where it doesn't reduce image quality.
-James
Yuval Levy
> WOW! We have a new programming kid on the block!
not so fast. I am just "hacking around" (and I hope to find the time to
write in my blog a "hacking Hugin for dummies" kind of article).
> Well done.
thanks.
> It compiles and works fine as well on OSX, both via a cmake build and an
> XCode build.
> It seems a bit faster but I don't have big projects at hand.
good, now I can continue improving it :-)
Yuv
James Legg
> On Fri, 2009-10-02 at 23:57 -0400, Yuval Levy wrote:
> > Hi all,
> >
> > One thing that bothers me about Hugin is how slow it is in opening the
> > fast (pun intended) preview. Start Hugin. Load a project. Click on the
> > fast preview icon and wait. Wait. Starr at the status bar with the (not
> > yet localized ;-) ) "Loading image:" messages, while one image after the
> > other is loaded. And the Ubuntu equivalent of the sandglass trying to
> > hypnotize me...
> >
> preview where it doesn't reduce image quality.
With the attached patch the fast preview uses the small images. If they
are smaller than the texture it wants to produce, it uses the large
image as it did previously instead.
This means:
1. Loading one preview speeds up loading the other.
2. Loading the fast preview speeds up display of the image previews
in the images tab, just like the traditional preview.
3. If Yuv's patch is modified to generate the small images, the
fast preview will (in general) display much faster with large
projects. The traditional preview would be faster if the small
images were loaded in the background too.
4. The fast preview does not loose any more image quality than it
did previously.
5. Without having the small images cached to start with, loading
the fast preview is a little slower, as it has to scale images
twice instead of once.
-James
Lukáš Jirkovský
2009/10/3 James Legg <lanky...@gmail.com>:
also reduce memory usage quite a lot because big images are in fact
used only when there are only few images.
But I think loading images in separate thread (with low priority) may be better.
Lukas
Yuval Levy
first of all my apology for replying so late. I needed to do more
detailed tests to be sure of my statements.
second, let me reassure you, in case it is necessary, that my mail, and
my patch, are not about shortcomings of the Fast Preview. Your Fast
Preview is one of the best features ever added to Hugin. A lot of the
great responsiveness of Hugin is to your credit.
That said, there are parts of Hugin, notably the Loading and the Image
Cache, that come *before* the Fast Preview and make the experience less
enjoyable than it could be, in my opinion.
James Legg wrote:
> On Fri, 2009-10-02 at 23:57 -0400, Yuval Levy wrote:
>> Hi all,
>>
>> One thing that bothers me about Hugin is how slow it is in opening the
>> fast (pun intended) preview. Start Hugin. Load a project. Click on the
>> fast preview icon and wait. Wait. Starr at the status bar with the (not
>> yet localized ;-) ) "Loading image:" messages, while one image after the
>> other is loaded. And the Ubuntu equivalent of the sandglass trying to
>> hypnotize me...
let me be clear: The waiting is not for the Fast Preview. It is the same
Waiting as when I go to the control points tab and the images are loaded
for the first time; or the crop tab; or any tab that shows the images.
> This doesn't help me for large projects.
I knew there were trade-off. I implemented an aggressive caching policy,
to trade off some memory, disk access, battery drain in exchange for
user time.
My findings are that performance is at least equal to an unpatched
version in my test cases.
> Big images take a long time to load. Since you do this in the idle loop,
> the UI actually gets less responsive.
Yes - this is bad design of my implementation. Using a separate, low
priority thread; as you and Lukas suggest, would be the right thing to
do. Bear in mind that the writer has zero formal training in computer
science and is just dipping the toes into C++. I welcome any help /
improvement from experts like yourself.
The way it is implemented now, I can interrupt the loading of the images
during the idle loop after every image. Which, I agree, makes it choppy.
When I load TIFF files over the network the choppiness is worse than
when I load JPEG files from the local hard disk; and in both cases I
would prefer a background thread, a smoother feeling, and the Fast
Preview available from the beginning of the loading process with
placeholders instead of the actual images, filled progressively.
But even the choppiness is less disturbing to me. Usually I start the
Hugin GUI loading a project that was previously set up via a batch run
on the command line with autooptimiser. It's equivalent to using the
Assistant's step 1 and 2.
After loading the project, I head out to either the CP-panel to set up
vertical control lines because the autooptimiser can not (yet)
automatically guess where is up and where is down; or to the Fast
Preview (I'm an instant gratification guy, and the Fast Preview is as
addictive as sugar for me).
In both cases, I have to wait for the Image Cache to do it's job - on
two images if I go CP-panel; on all images if I go Fast Preview.
Even worse with the CP-panel: when I switch to the next images pair, I
have to wait for one additional image to load. This is the kind of time
I want to save.
My findings are:
* for my typical 360°x180° full spherical, six fisheye shots panorama,
the aggressive cache loads everything into RAM and clicking through to
the next image pair on the CP tab is faster; accessing both the CP tab
or the Fast Preview is equally fast or faster than withouth my
aggressive caching.
* for larger projects that do not fit in memory (2GB on my dynobook, 8GB
on my workstation), the CP tab is faster; the Fast Preview will take as
much time without the cache.
> Ideally, one would solve this problem by loading the images in another
> thread with a lower priority.
as I said, it is just a proof of concept and a wet toe into C++ -
improvements are welcome, and I am sure that a crack like you can get
much better performance out of my rough quick and dirty fix.
> Another issue I see is that it eats memory. If I load a project where
> all the decoded images don't fit into RAM, this makes my whole system
> unresponsive as it eventually it starts moving stuff I'm trying to use
> to swap memory.
While I have experienced that it eats memory, and I expected there to be
trade-offs, I've not experienced it that dramatically as you describe below.
> If I try to load the fast preview while this is going on, the following
> happens:
> * The preview gets the first image from the image cache. It is now
> loaded, where it wasn't before, saving a file load.
> * The preview tries to free up some space, in case there are more
> images in the cache than fit the allowed memory. There is, so
> the image cache frees most of the images that have been
> preloaded.
> * The preview tries to get the second image from the image cache.
> There is a high probability it was freed by the last step, so it
> loads it off the disk again.
> * The preview again asks the image if it would like to free some
> memory. It is likely to free the image accessed the earliest,
> which is another preloaded one.
> * ...
> So it only saves one file load if left too long to preload images.
I "interrupted" the loading at different stages and did not experience
that chaos. The Image Cache shows nicely in the status bar what is it at
exactly. I've used a 40 images project, and interrupted it at 10; 20;
30; "pre-loaded" images with a click on the Fast Preview button. In all
cases, I have counted only 40 changes in the status line, means nothing
has been loaded twice.
But given how it is implemented I can imagine potential conflicts, it
depends a lot on the machine's specs; the project at hand; the user's
behavior. Hence I think that it is best:
(a) improved with a low priority thread
(b) added as a preference - or (c) kept as a private patch.
I am currently not proficient enough to do (a), and I don't know how
much time it will take. I'd appreciate any help, pointers, guidance. I
know the conpcepts, I just lack the practical C++ experience.
For the choice between (b) and (c), it is up to the community. I can
keep the patch here, and apply it privately to my instance of Hugin. I
did publish it here because it may be useful to others; and because I
believe that the next thrust of development should go into improving the
user interface, making it more responsive and bring it up to par with
the Fast Preview. I want fast, predictive pre-loading. I want to use all
the memory and all the power that the machine can yield. Others may
prefer it to be more economical (and who knows, maybe I'll find myself
on battery power and will switch the aggressive caching off too).
> I have some suggestions to get around these issues:
> 1. Load images in a low priority thread separate to the GUI thread
> if possible.
yes, please, do!
> 2. Load the small images from the small image cache. These are not
> freed, as they occupy a small amount of memory each. They are
> used in the images tab and the traditional preview.
I've tried your patch on my dynobook. It's 1.6GHz 2GB RAM and fast
preview is always maximized to occupy the whole 1400x1050 high
resolution display. It was actually slower, and I saw on the status bar
the Loading and Resizing messages running.
My explanation is that even to use the small images, the full sized ones
need to be loaded. so resizing is an extra cost.
My observations are for both a 6 pictures and a 40 pictures project. I
should try with a 300 pictures project but did not have the time.
And in best multi-tasking tradition (bad habit!), I tried the 40
pictures project while building Hugin in the background (better leave
the machine alone and do other things in the meantime). No problems.
> I wrote the fast preview to use the large image cache to create the
> textures it uses. This is so that if you are, for instance, just
> remapping an equirectangular pano to a little planet, the image is
> sharper as it knows it can allocate more texture memory to it.
I love the Fast Preview. It is sharp and clear on the 1400x1050 of the
dynobook and on the 1920x1200 of the workstation.
> We could make the fast preview use the small image cache for the smaller
> textures so that the above routine has a better effect on performance of
> displaying the fast preview: All the textures could be derived from the
> small image cache on a sufficiently large project.
I tried your patch and I did not feel any improvement. Because the
bottleneck, at least on my settings, is not the Fast Preview. It's disk
access to the image files.
I welcome any improvement in the cacheing. The current cacheing is very
basic and waits for the user to ask for something before providing it. I
want a more proactive cacheing, and my patch was just an initial proof
of concept. I shared it. It is good enough for me. The question is:
should I keep it for myself and those who have downloaded and tried the
patch? or should it be made available for everybody, in binary form?
How would you feel if I add this as a preference (disabled by default)?
Others with more C++ experience than me can improve on it with the low
prio thread and smarter caching strategies.
Yuv