Delivering a large image collection

[Stefano Cossu]

Hello,
I am evaluating an image server for delivering our Museum image colleciton (~80,000 images and growing) over the IIIF protocol. So far I have looked into IIPImage (which we already run for a smaller project) and Loris.

Assuming that this collection will have a few (hundreds) top hits and a vast majority that is much less frequently accessed, I am considering both performance and good cache management as the main factors.

I have a few questions, some of which I have recently discussed with the IIIF developers and I am sharing here:

• Can someone point out some comparably large image collections that are currently being delivered over IIIF?
  
• It seems like IIPImage is very fast but it only relies on Memcached; on the other hand, Loris is not as fast but it has a disk-based cache that seems more appropriate for my case. Is IIPImage planning on implementing a disk-based cache in the near future?
• What is the advantage of having a built-in cache versus putting Squid or Varnish in front of the image server?
• How come IIPImage uses the Kakadu SDK, involving annual license fees, to deliver JP2, while Loris just needs the free Kakadu binaries? Is this a factor responsible for the better performance of IIPImage?
• I am open to other solutions if someone can point out a valid alternative.
  

Thanks,
Stefano

--

Stefano Cossu
Director of Application Services, Collections

The Art Institute of Chicago
116 S. Michigan Ave.
Chicago, IL 60603
312-499-4026

[Michael Appleby]

Yale is hosting ~40,000 images for the Yale Center for British Art as
well as ~30,000 images related to digital manuscript projects. Our
infrastructure is based on Djatoka and homebrew IIIF shim. We run two
load-balanced images servers in Amazon (i.e., AWS EC2).

We use separate disk caches for JP2s and JPEG tiles on the server, as
well as a Varnish for in-memory cache. The advantages to this are (1)
we can allocate more cache to tiles on disk than in memory and (2)
fetching JP2s from our central store (AWS S3) has overhead.

Mike

--

Michael Appleby
Associate Director
Academic Software Development
ITS Campus Community Technologies
Yale University

e michael...@yale.edu
p 203.436.9354

[Tom Crane]

Hi Stefano,

speaking on behalf of the Wellcome Library...

We have 6m+ JP2s (growing at 50,000 a day) and serve tiles generated from IIPImage. The public aren't always seeing IIIF endpoints (they will do very soon, once the Auth spec is ready) but they are still seeing IIPImage-generated tiles, so for the purposes of this discussion it makes no difference.

We do not use the memcached feature of IIPImage, we cache generated tiles at a reverse proxy a bit higher up (in this case Microsoft ARR but exactly equivalent to Varnish in this context) because we meddle with various HTTP headers on the way out. I think I would always prefer to do it this way and keep the IIPImage instances completely "vanilla". We can store a lot of generated tiles like that!

We have two IIPImage servers handling the public traffic at any one time, these are VMs equivalent to 4-core, 16GB RAM. These have been ample so far.

We also maintain a large JP2 cache mounted on the IIPImage boxes as there is overhead in retrieving JP2s from Wellcome's DAM. Layers above the IIPImage instances ensure the JP2 is present in the cache before a request makes it through to IIPImage.

IIPImage requires a Kakadu licence to build* because it builds against the Kakadu binaries, therefore uses Kakadu in-process rather than out-of-process like Djatoka or Loris. This should make it faster.

Here's a blog post about compiling IIPImage with Kakadu by my colleague Adam Christie:

http://circle-theory.blogspot.co.uk/2015/01/jpeg2000-support-with-iipimage-iipsrv.html

Tom

*If you have pyramidal TIFFs rather than JP2s you can build IIPImage directly without the Kakadu dependency.

[Jon Stroop]

Stefano,
We chatted a bit in person yesterday, but to answer some of your questions that didn't come up: We're using Loris to serve just over a million images from various systems in our library. Similar to the situation you describe, there are some high-traffic items, a lot of lower-traffic items, and also items that are high-traffic for short periods of time. For example, we put the manuscript of The Great Gatsby online about a year ago and there was a press release in the New York Times that gave us a burst of traffic for a three or four days.

The press release case is where I think having Varnish in addition to a filesystem cache is helpful. Most of the time, the filesystem cache is just fine, and unless your images are huge or you're getting a lot of traffic, you can keep it quite small. This instance:

http://libimages1.princeton.edu/osd-demo/?feedme=pudl0123%2F8172070%2F01%2F00000001.jp2

does not have Varnish installed, and has 2 GB dedicated for caching on the file system.

If you wanted to use Varnish only and forgo the filesystem cache completely, you can turn caching off in Loris's config file:

https://github.com/loris-imageserver/loris/blob/development/etc/loris2.conf#L27

I've never done that in a real production system, but it is one of the reasons the option exists.

As far as performance goes, I think you'd be fine w. either IIP or Loris. I'm sure having Kakadu compiled into IIP does give it an edge, but once things are cached, I doubt there's much or any difference, especially for the size of your collection and if you're willing to dedicate some storage and/or memory to caching.

Finally, note that nothing from Kakadu (binaries or SDK) is "free". From http://kakadusoftware.com/downloads/ :

You are free to trial these executables and even to re-distribute them, so long as such use or re-distribution is accompanied with this copyright notice and is not for commercial gain. Note: Binaries can only be used for non-commercial purposes.

I think the key word is trial.

-Jon

--
-- You received this message because you are subscribed to the IIIF-Discuss Google group. To post to this group, send email to iiif-d...@googlegroups.com. To unsubscribe from this group, send email to iiif-discuss...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/iiif-discuss?hl=en
---
You received this message because you are subscribed to the Google Groups "IIIF Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to iiif-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Stefano Cossu]

All,

Thank you very much for providing use cases. From the variety of implementation scenarios I can deduce that the choice of an image server is less critical to performance than the caching and infrastructure strategy. 

Also thanks for pointing out the cache layer for retrieving JP2. This will concern us too. Definitely we can avoid Kakadu licensing fees by using PTIFFs, but that would likely be counterweighted by storage and I/O costs. 

Stefano Cossu

Director of Application Services, Collections

The Art Institute of Chicago

116 S Michigan Ave.

Chicago, IL 60603

+1-312-499-4026

sco...@artic.edu

[Matt McGrattan]

• Can someone point out some comparably large image collections that are currently being delivered over IIIF?
  
• It seems like IIPImage is very fast but it only relies on Memcached; on the other hand, Loris is not as fast but it has a disk-based cache that seems more appropriate for my case. Is IIPImage planning on implementing a disk-based cache in the near future?
• What is the advantage of having a built-in cache versus putting Squid or Varnish in front of the image server?
• How come IIPImage uses the Kakadu SDK, involving annual license fees, to deliver JP2, while Loris just needs the free Kakadu binaries? Is this a factor responsible for the better performance of IIPImage?
• I am open to other solutions if someone can point out a valid alternative.
  

The Bodleian currently has around 100-150,000 images available via IIIF on our Digital.Bodleian service which is currently in beta but the public soft launch will be in the next week or so, and the full launch in early July. So I'll share the URLs very soon. 

Everything that we digitise from now on in our new workflow will be available automatically via this service, also, and we are migrating our existing image archive [approx. 3,000,000] over the 6-12 months.

We use IIP with Kakadu compiled in, and we have a license for Kakadu. Incidentally, our experience with Kakadu exactly mirrors Adam Christie's. We had to disable the SSSE3 and AVX2 optimisations in order to stop KDU core dumping. It took a _long_ time for me to work out why KDU 7.x was crashing on our systems but KDU 6.x was not. 

We use a Varnish cache sitting in front of everything caching all of the image tiles. We also store the info.json responses separately in Redis [although we could and may use something else].

Much like the Wellcome, our IIP installs are vanilla, but the image servers also run a resolver/rewriter application which serves a few functions:

i) resolving item UUIDs to absolute file-paths on our backend repository storage. 

ii) intercepting info.json requests and serving them from Redis.

iii) changing the content of the info.json if necessary. Currently the info.json exposes the full absolute file-path, but we may decide not to do that.

iv) handling interaction with an authentication/authorisation service as and when we are ready to implement that.

We have currently have 3 VMs running IIP and the resolver application, for the equivalent of around 8 total CPU cores.

However, we find that the limitation isn't the performance of the VMs. Even under heavy load testing, the load average on the image servers stays pretty low because -- at least in our infrastructure -- disk and network i/o between the image servers and storage is the limiting factor.

Using Varnish or a similar cache is where the real performance gain is for us, at least. To make best use of that, I/we pre-warm the cache with a cron job that periodically grabs thumbnails and the info requests [iiif, deepzoom, and iip protocol, as we use all of these] for everything in our index.

Where this really helps is with manifest generation, as making info.json requests in order to create canvas information is multiple orders of magnitude faster via Varnish & Redis than trying to get the info from the images via Kakadu, or from IIP.

Thanks!

Matt

Collection Delivery Architect

Bodleian Libraries, Oxford.

[Stefano Cossu]

That sounds interesting. I am looking forward to seeing the new Bodleian service when it launches.

I have a couple more questions:
- What do your Varnish cache timeouts look like? If I have, say a 24h timeout that seems reasonable for the few high-traffic images, but there are also many (10s, 100s) thousands of other images that get accessed maybe once a week each. Cumulatively these images would create thousands of new caches every day. Is that still OK?
- Do you advise having static thumbnails rather than dynamically generated ones? Thumbnails are a slightly different category of images IMO due to the way they are created and accessed.

Thanks,
Stefano

Stefano Cossu
Director of Application Services, Collections

The Art Institute of Chicago
116 S. Michigan Ave.
Chicago, IL 60603
312-499-4026

[Tristan Roddis]

For the Qatar Digital Library (www.qdl.qa) we host and serve around 500,000 JP2 master images using IIPImage (not IIIF-compliant yet, but working on it!).

We use a static version of IIPImage with Kakadu compiled in. We don't use memcached, but we do have Varnish in front of IIPImage, with a short timeout of 5 minutes.

We run this off two "m3.large" EC2 servers, but set to autoscale up to four servers if CPU load is consistently high. All IIPImage cluster members access the source JP2 files from a shared GlusterFS storage cluster.

So far, the whole thing has worked pretty much seamlessly. The only trouble we ran in to was some strange one-second delays in serving images when the site was under high load. This turned out to be due to us using the Apache module mod_fcgid instead of mod_fastcgi, and this went away once we switched to mod_fastcgi.

Hope that helps,

-Tristan.

--
cogapp

Tristan Roddis
Head of Web Development
+44 1273 821 600
www.cogapp.com
 
 
News 
Qatar Museums’ website recently won an international award for Best Cultural Experience. Find out more here.
For more Cogapp news, please follow us on Twitter and LinkedIn

--

[Stern, Randy]

In terms of images available through IIIF, Harvard is committed to using Mirador to replace our existing page turner, but are also experimenting with other image sources as well.  We currently have on the order of 6,000,000 still images available through the IIIF Image API, and on the order of another 10,000,000 or so page turned document page images available through a Beta version of Mirador on top of the IIIF Presentation/Image APIs.  We expect to look at IIP Image and Loris for the future, but are currently delivering images using a IIIF Image API Shim atop a Luratech ICS based engine. We cache static thumbnails for our page delivery service and to improve performance in our catalogs, but are currently not caching ordinary image delivery.  We need to do that, though ;-)

[Tom Crane]

Hi all,

Matt and I have both mentioned keeping the image server "vanilla" and I think this will be important as the Image API acquires new features (like auth) that image server software may choose to ignore or not provide. Even though the image server can generate the info.json, a lot of implementations probably won't serve that exact info.json up to the internet - they'll store it in Redis, they will modify it, change @ids, they will add services (like login and logout). It will start getting complicated if IIPImage, Loris et al have to keep up with auth specifications and try to enforce auth in the image server. An institution that has a need for auth and the means to supply the auth services will also have the means to have logic in front of their image servers to serve up modified info.json, and to authorise tile requests.

Personally speaking I would prefer it if my image server assumes an open world in which nothing is protected and no declarations of auth services need to be made, and add these features in my institution-specific layer above the "raw" IIIF Image API. The end user doesn't see the info.json that IIPImage gave me, it sees my edited version.

That means I can easily switch between IIPImage, Loris and others without worrying about how they integrate with my specific auth implementation.

Tom

[Andrew Hankinson]

Slightly off topic, but perhaps useful: I think we worked out the problem with SSSE3 and AVX in Kakadu 7.5+ for IIP.  I don't know if this is the same problem you were experiencing, though. For us it wasn't core dumping, but simply refusing to compile.

The pull request, with an explanation, is here: https://github.com/ruven/iipsrv/pull/38

I can confirm that it's working for us on both Mac and Linux systems with the latest version of IIP

[Adam Christie]

Hi Andrew,

That looks really useful - thanks! Yes - I too found that it simply wouldn't compile as received. I'll see if I can do some experimentation with that soon. Will update the circle-theory article if I can get that working.

Adam.

[Ruven]

Hi Stefano,

I'm Ruven, the IIPImage maintainer.

To answer your question about disk-based caching, IIPImage has in fact 2 levels of cache: an internal cache for raw image data and memcached integration for storing processed tiles or output metadata. Yes, these are both RAM-based caches, rather than disk-based. However, the memcached cache has the nice property of allowing your front-end server (NginX and maybe others) to share this same cache and thereby take these processed IIIF tiles without even needing to make a request to the image server.

The best overall solution, is to combine multiple types of cache and, as already pointed out, using Varnish in front of your server can make a big difference. I use both Varnish and Memcached on the IIPImage demo server and this combination has comfortably resisted a couple of Slashdottings without problems.

For Kakadu integration, as Tom has pointed out, compiling against Kakadu allows fast in-process access to the Kakadu decoder, which will always be faster than making external calls as would need to be done if just using the Kakadu evaluation binaries. On the issue of JP2 vs TIFF, don't forget also that TIFF is considerably faster to decode than JPEG2000 even with Kakadu and that if storage space is an issue, it's possible to use JPEG tile encoding within the TIFF itself allowing good compression if you're willing to surrender image quality.

For large collections using IIPImage in the museum world, the NGA have their entire collection online using IIPImage and I have 350,000 images in our system at the Louvre.

> What do your Varnish cache timeouts look like? If I have, say a 24h timeout that seems reasonable for the few high-traffic images, but there are also many (10s, 100s) thousands of other images that get accessed maybe once a week each. Cumulatively these images would create thousands of new caches every day. Is that still OK?

The timeout will depend more on whether your images change often or not and I would imagine they don't. Of course if you end up also using Varnish for other website content, then this is what will probably determine your timeout. For the quantity of cached items, with Varnish you simply set a max cache size, so the total number of items in the cache shouldn't be an issue.

 > Do you advise having static thumbnails rather than dynamically generated ones? Thumbnails are a slightly different category of images IMO due to the way they are created and accessed.

Small thumbnails don't take up much space and are relatively quick to generate, so it's easy to pre-generate these statically if you want the fastest possible speed. However, if you have several sizes of thumbnail or other intermediate size views, then it'll probably make more sense for the image server to generate the intermediate sizes.

Cheers,
Ruven

-----------------------------------------
http://iipimage.sourceforge.net
-----------------------------------------

[Stefano Cossu]

Ruven,
Thanks very much for the information. See replies in line below.

Stefano Cossu
Director of Application Services, Collections

The Art Institute of Chicago
116 S. Michigan Ave.
Chicago, IL 60603
312-499-4026

On 6/18/15 4:57 PM, Ruven wrote:

Hi Stefano,

I'm Ruven, the IIPImage maintainer.

To answer your question about disk-based caching, IIPImage has in fact 2 levels of cache: an internal cache for raw image data and memcached integration for storing processed tiles or output metadata. Yes, these are both RAM-based caches, rather than disk-based. However, the memcached cache has the nice property of allowing your front-end server (NginX and maybe others) to share this same cache and thereby take these processed IIIF tiles without even needing to make a request to the image server.

The best overall solution, is to combine multiple types of cache and, as already pointed out, using Varnish in front of your server can make a big difference. I use both Varnish and Memcached on the IIPImage demo server and this combination has comfortably resisted a couple of Slashdottings without problems.

If I have >4Gb pyramidal TIFFs (I do indeed, although very few), how will memcached handle it? Shall I load my server with RAM?

For Kakadu integration, as Tom has pointed out, compiling against Kakadu allows fast in-process access to the Kakadu decoder, which will always be faster than making external calls as would need to be done if just using the Kakadu evaluation binaries. On the issue of JP2 vs TIFF, don't forget also that TIFF is considerably faster to decode than JPEG2000 even with Kakadu and that if storage space is an issue, it's possible to use JPEG tile encoding within the TIFF itself allowing good compression if you're willing to surrender image quality.

This is quite interesting. I had not thought that JPEG-encoded PTIFFs can be processed without Kakadu. Does that apply only to the tiles or can it also be used for the full image?

You might have noticed a large thread discussing the use of OpenJPEG vs. Kakadu on this same board. Has this option been brought up there?

--

[David Beaudet]

At the NGA, we have installed varnish in front of IIP but we have Apache fronting it to provide TLS since varnish is unlikely to ever ship with a TLS layer. Interestingly, we don't have much of a cache size configured. The current varnish install is more like a proof of concept since we hadn't previously used it. We do expect to beef this up in the near future but even without varnish, IIP has had no problems keeping up with load.

We use standard jpeg (not jp2) encoded TIFF files at 90 quality with good results, even at >30k resolution. The challenge for us with high resolutions is the CPU requirements for generating the PTIF images in the first place, but fortunately that's an infrequent activity.

General question: if we shifted from PTIF with standard jpeg in favor of jp2 compression would we see a performance impact, either good or bad? I had heard at one point that jp2 resize operations are supposed to be a lot faster than standard jpeg due to the way the wavelets are arranged, but I have not run any tests. I'm also not sure that resizing operations are even performed by IIP during the normal course of serving zoom tiles because it's operating off resized tiles embedded in the PTIF already right?

[Andrew Hankinson]

We use NginX with round-robin load balancing on five IIP processes, and a memcached store on top of that. With that we achieve acceptable response times, but we're not serving large amounts of data to a lot of people.

In our experiments, JP2 is always slower/more resource intensive than PTIF (in any compression) in IIP. My guess is that opening and decoding the images takes quite a bit longer, so while discrete operations like rotate may be faster (I don't know if they are), any speed advantages will be lost in the decoding step.

[beaudet]

Andrew,  are you using the commercial libraries or open JPEG for JP2 support with IIP? 

Thanks, 

Dave Beaudet 

National Gallery of Art 

d-be...@nga.gov 

bea...@gmail.com 

-------- Original message --------

From: Andrew Hankinson <andrew.h...@gmail.com>

Date:06/19/2015 10:03 AM (GMT-05:00)

To: iiif-d...@googlegroups.com

Cc:

Subject: Re: [IIIF-Discuss] Re: Delivering a large image collection

You received this message because you are subscribed to a topic in the Google Groups "IIIF Discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/iiif-discuss/iJGsVv1Sd48/unsubscribe.
To unsubscribe from this group and all its topics, send an email to iiif-discuss...@googlegroups.com.

[Andrew Hankinson]

We're using Kakadu and compiling IIP against it.

[Ruven]

> If I have >4Gb pyramidal TIFFs (I do indeed, although very few), how will memcached handle it? Shall I load my server with RAM?

Memcached doesn't load the image itself into RAM, only the requested tiles that IIPImage outputs, so you can use whatever level of RAM you want. By the way, what kind of images are these? Xrays I presume? IIPImage comfortably scales up to this size and well beyond. The largest image I've used IIPImage with is a 1TB image ;-)

> I had not thought that JPEG-encoded PTIFFs can be processed without Kakadu. Does that apply only to the tiles or can it also be used for the full image?

This is classic JPEG encoding, not JPEG2000! This is used to compress each TIFF tile, so the whole image at all available resolutions is effectively compressed in this way.

> I had heard at one point that jp2 resize operations are supposed to be a lot faster than standard jpeg due to the way the wavelets are arranged,  but I have not run any tests

Yes, JP2 is able to use progressive encoding with an ordering you define. So, you can order internally by resolution level or color component etc. This kind of ordering plus the use of codeblocks and precincts can greatly help improve decoding speed. However, even at it's best, JP2 can be several times slower than decoding tiled multi-resolution TIFF, simply because JP2 decoding is much more complicated than for TIFF.

Cheers,
Ruven
-----------------------------------------

http://iipimage.sourceforge.net
-----------------------------------------

[Jeremy Echols]

Probably should have weighed in before, but I think it's important to consider traffic a lot more than the quantity of images.  The Historic Oregon Newspaper (http://oregonnews.uoregon.edu/) site has over 600,000 images that are generally around 30 megapixels each.  The whole thing is running on a single server that includes RAIS, the Django site, Solr, MySQL, and Apache.  We cache all thumbnails just via Apache, but we don't do any tile caching (it isn't a feature we've had time to build into RAIS).  Generally speaking, tiles return in under 250ms, but it's not hard to do some fake load testing to push that up to 25 seconds per tile.

But there are a few things going on here:

- We're very low-traffic - we serve just over 100,000 tiles a day.  It's pretty unlikely we'd survive "the slashdot effect".
- RAIS is fairly fast *for tiled JP2s*.  It's almost unusable for large JP2s that weren't encoded with tiles.

If you expect more traffic, even a 5000-image site will need to look at a different solution than we have in place.  Additionally, if you have pyramidal TIFFs available, and don't mind the extra disk space, IIP will undoubtedly crush just about any other option.

[Jeremy Echols]

I sort of lost my point here - if you don't expect a lot of traffic, you can probably just stand up any server and see how things go.