Bench mark software to compare performances of image servers

[Tao Zhao]

Hi there,

I want to compare the performances of IIPImage, Loris, and Cantaloupe. Is there a benchmark available to test or any other approaches to adopt? Our images are primarily 200MB - 2 GB TIFF files.

Thanks,

Tao

[Matt McGrattan]

When I was doing this, I did it two ways:

Via a simple Python script that pulled a random list (of configurable size) of images from our backend, and then (sequentially) requested thumbnails, and some random 512x512 tiles for each, logging elapsed time and average response time per request.

JMeter/Blazemeter type tests, hitting the whole stack with hundreds or thousands of concurrent requests, testing how they held up.

I didn't publish any results, though. We are building some containers for OpenJPEG based stacks, too, so I'll try to publish some numbers when we get that done.

Matt

[Tristan Roddis]

I haven't got any specific benchmarks or scripts, I'm afraid, but only some general advice:

1) Benchmarking tools
Apache Benchmark is great for spot tests, and is very configurable, including launching multiple threads. It should be installed on any system that has Apache (command line: "ab"). For more involved tests, JMeter is great, and can do advanced things like read in URLs or parameters from CSV files, but it's somewhat confusing to configure.

2) Caching
For each of the systems you mention, you may want to test them with and without caching (or with different caching parameters). E.g. IIPImage can optionally use memcached, and I for one would be very interested to know how much difference this makes in practice.

3) Image formats
Petr Pridal mentioned at the Ghent meeting that the encoding options of source JPEG2000 files can affect the performance of underlying decoders (Kakadu versus OpenJPEG), so you may also want to try varying the encoding options, or even file type (JP2 versus pyramidal TIFF) to see if that makes any difference. Unfortunately, there was a lot of Belgian beer involved at the time, so I can't remember the specifics. Petr, can you comment?

-Tristan.

--
-- You received this message because you are subscribed to the IIIF-Discuss Google group. To post to this group, send email to iiif-d...@googlegroups.com. To unsubscribe from this group, send email to iiif-discuss...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/iiif-discuss?hl=en
---
You received this message because you are subscribed to the Google Groups "IIIF Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to iiif-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

cogapp

Tristan Roddis
Head of Web Development
+44 1273 821 600
www.cogapp.com
 
 
News 
Qatar Museums’ website recently won an international award for Best Cultural Experience. Find out more here.
For more Cogapp news, please follow us on Twitter and LinkedIn

[Matt McGrattan]

Ruven Pillay did testing around JP2 encoding for performance, in the context of IIP, and has presented that at a few events/conferences.

See the last few slides in this:

http://www.dpconline.org/component/docman/doc_download/1358-2014-nov-jp2k-ruven

On top of that, OpenJPEG based systems definitely prefer (for huge values of prefer) tiled images, rather than single tile images, for example.

Matt

On Friday, 11 December 2015 11:13:50 UTC+1, Tristan Roddis wrote:

I haven't got any specific benchmarks or scripts, I'm afraid, but only some general advice:

1) Benchmarking tools
Apache Benchmark is great for spot tests, and is very configurable, including launching multiple threads. It should be installed on any system that has Apache (command line: "ab"). For more involved tests, JMeter is great, and can do advanced things like read in URLs or parameters from CSV files, but it's somewhat confusing to configure.

2) Caching
For each of the systems you mention, you may want to test them with and without caching (or with different caching parameters). E.g. IIPImage can optionally use memcached, and I for one would be very interested to know how much difference this makes in practice.

3) Image formats
Petr Pridal mentioned at the Ghent meeting that the encoding options of source JPEG2000 files can affect the performance of underlying decoders (Kakadu versus OpenJPEG), so you may also want to try varying the encoding options, or even file type (JP2 versus pyramidal TIFF) to see if that makes any difference. Unfortunately, there was a lot of Belgian beer involved at the time, so I can't remember the specifics. Petr, can you comment?

-Tristan.

On 10/12/2015 15:47, Tao Zhao wrote:

Hi there,

I want to compare the performances of IIPImage, Loris, and Cantaloupe. Is there a benchmark available to test or any other approaches to adopt? Our images are primarily 200MB - 2 GB TIFF files.

Thanks,

Tao

--

-- You received this message because you are subscribed to the IIIF-Discuss Google group. To post to this group, send email to iiif-d...@googlegroups.com. To unsubscribe from this group, send email to iiif-discuss+unsubscribe@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/iiif-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "IIIF Discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to iiif-discuss+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jon Stroop]

FWIW, Loris supports TIFF, but doesn't do anything special with them (or w/ PTIFFs), so you're not likely to get very good performance with TIFFs unless you're willing to do a lot of caching. That said, you have two caching options (and can use both). 1) by default there is a filesystem cache turned on that will cache your derivatives to the file system 2) you can use any front-end cache (e.g. Varnish) to avoid hitting the server altogether whenever possible. Loris gives you the option to turn the file system cache off altogether and just let a front-end cache handle your caching.

I'm not sure about Cantaloupe's handling of TIFFs / PTIFFs, but I expect that IIP is going to give the the best performance if that's your source format.

-Js

--
-- You received this message because you are subscribed to the IIIF-Discuss Google group. To post to this group, send email to iiif-d...@googlegroups.com. To unsubscribe from this group, send email to iiif-discuss...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/iiif-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "IIIF Discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to iiif-discuss...@googlegroups.com.

[David Beaudet]

At the IIIF hackathon in Philadelphia, I had hoped to, but didn't have time to, perform the testing you're about to do.  I did manage to acquire a Kakadu license free of charge for the purpose of performing this testing with JPEG2000 images, so if you don't already have a license but might consider purchasing one, you might want to ask them for an evaluation license so you can compile IIPImage and other servers against Kakadu.

I did notice that when using the ab tool against IIPImage, that it seemed to be slower at IIIF requests than native IIP image requests, but that's about as far as I managed to get with my tests.  As others have already mentioned, there are many variables in play here.  Caching at multiple levels, formats of source images, resizing algorithms (IIP supports two), commercial vs. open source image libraries, etc.

Please do post your testing methodology and results to the list.  I imagine it would be really helpful to have this data published to IIIF.io as well.

[Tao Zhao]

That'd great that you post the results.