tn/*.html generation very slow for large image numbers

[Frank]

Hi,

I would like to apply album to a sequence of images which are produced
once per second over a period of 3 hours.
So I have more than 10000 images but I noticed, that the generation of
the .html files in the tn/ folder gets very slow for large image
numbers.
Has this problem been noticed before?
However, it would be very important for me if this could be speeded
up.

Frank

[album author - David Ljung Madison]

The fact is that creation of thumbnails takes time.

Fortunately it won't need to do it again the second time (although it
needs to check the timestamps to verify this, so if you have tens of
thousands of images in a directory it will still take some time, it
might make sense to break those up - do you really want an HTML page
with 10000 thumbnails?)

There are a few things you can do to speed it up.

1) Make sure you aren't using "medium" sized images, these will more
than double your time for handling each new image.
2) Try a different program to scale the images, for example the album
plugin images/epeg is generally about 3x faster for scaling images.
To use it, make sure you have plugins installed and in the right path,
and you can do "album -plugin
images/epeg

[Frank Breitling]

Thanks for your fast answer.
I will try out your suggestion for speeding up thumbnail creation.
However my problem appears after all thumbnails have been created. Only
the .html pages are still being created at a much slower rate of about 1
per 10 seconds.
Maybe the slow down has something to do with the long directory contents.

Frank

[Frank]

It is the generation of Indexes which takes so long.
I don't understand why but at this rate if will take at least another
week to finish.
There is a serious problem in the code that needs to be fixed.

Frank

[album author - David Ljung Madison]

If you are using a theme, then it can take almost a second per index
page on your run of the mill computer.  It shouldn't take 10 seconds,
unless you're running some theme that's doing lots of work.  Does this
take the same amount of time per html if you divide it down to 100
images or so?  And is this happening even if you try to regenerate the
album?  It shouldn't generally need to regenerate most of your html
files if nothing has changed.
Two things that can help is to use -notheme, or else to try the images/
xy_cache plugin.  But it sounds like something else is wrong - if you
want to contact me directly we can try to debug it.

[Dave Hansen]

I don't think it's necessarily an album bug. I've got a decent size album:

$ ls | wc -l
7155

after I rm tn/*.html, a complete album run takes ~4 minutes. I got that
down to ~30s (with a several-year-old 1.6GHz Celeron) with some
modifications. For me, the slow speed was a combination of fork/exec
time and the sheer overhead of invoking 'identify' over and over. I
implemented an album plugin that uses native perl calls to replace
calling 'identify' and caches the results in an extended attribute. The
album plugin is quite simple, but the fetch_exif() is the real magic and
it's not really in sharable shape:

http://sr71.net/~dave/projects/album/native_perl_exif.alp.txt

You might also have some issues with your filesystem.
Multi-thousand-file directories can get a bit slow in some cases.

What is your user/system CPU split? Are you CPU or I/O bound?

[Frank Breitling]

David,

I can confirm that it is in fact the usage of a theme (like Black3,
Blue, Column, Stars) that causes this tremendous slow down.
Without a theme or the PopUp theme which doesn't generate the html files
album runs quite fast.
Moreover does the slow down increase not simply linearly with the number
of images but stronger (maybe exponentially). While 250 images took 45
seconds, 500 took 150 seconds. In the case of 40000 images one tn/*.html
file takes about 10 seconds.

I didn't notice any significant improvement by using the xy_cache
pluging but I didn't study this in detail.

Dave H.,

Your plugin sounds very interesting, but unfortunately I wasn't able to
run it.
My user/system for regenerating the tn/*.html files of 250 images was as
follows:

test$ time album -theme Black3
Read conf: ~/.album.conf

This is album v4.06 on linux

Read conf: test/album.conf
Images: test
[XXXXXXXXXXXXXXXXXXXXXXXXX]
Indexes: test
[XXXXXXXXXXXXXXXXXXXXXXXXX]

real 0m45.556s
user 0m43.910s
sys 0m3.130s

Without a theme it is

test$ time album
Read conf: ~/.album.conf

This is album v4.06 on linux

Images: test
[XXXXXXXXXXXXXXXXXXXXXXXXX]
Indexes: test
[XXXXXXXXXXXXXXXXXXXXXXXXX]

real 0m5.813s
user 0m0.490s
sys 0m0.530s

So in conclusion I believe this is a general problem of album using
themes, which should be reproducible on any system.

Please let me know if you need further details to trace it down and
please keep me posted if you find any solutions for this problem.

Frank

[album author - David Ljung Madison]

I've talked with Frank and looked into this issue and realized that
there is a throughput issue that grows as the number of images grows.
I've got a fix which will be in the next album release (v4.08) which
will also include a new plugin for ExectImage by Frank which does
speedups for image conversion (like epeg, which may not be easy to
install). As mentioned above, the xy_cache plugin can be a big
speedup if you ever re-run album on a directory.

If you want an early copy of the next album, let me know, otherwise it
should be out in a week or two.

On Dec 30 2011, 7:23 am, Frank Breitling <frank.breitl...@gmx.de>
wrote:

[CK]

Hey David,

Did v4.08 ever get released?   Or is there still any plan to do so?

[album author - David Ljung Madison]

My bad - I got caught up in other things.  Going through the release process now.