Islandora Performance

[aaron...@commonmediainc.com]

Hello -

We're seeing abysmal performance for authenticated users on our Drupal 7 islandora install (page load > 1 minute), not seen with authenticated users.  Aside from standard Drupal caching strategies (which have minimal effect on Auth users) are there any Fedora-side or islandora specific tuning options?

Specifically with fedora-side options, I'm wondering if I can enable caching of thumbnail images - or verify that any stock cache is enabled.  I'm inheriting this installation and can't vouch for any particular feature being enabled.  The bulk of our load times seems dedicated to serving thumbnails, although everything is generally slow as well.

Thank you!

[Nigel Banks]

Hello,

I'm not sure I understand could you please clarify "We're seeing abysmal performance for authenticated users"... "not seen with authenticated users"? 

I agree > 1 minute page loads is awful. Performance tuning can be a very specific exercise. Could you provide some number's to work from? You can use the networking tab in the developer tools for Firefox/Chrome/Safari to get the times required to fetch content. Also could you more detail about your install? The version numbers of Fedora / Islandora and what modules you have installed?

Nigel

[aaron...@commonmediainc.com]

Here is a screenshot of my network tab in chrome dev tools:

It looks like the thumbnails (highlighted item) is blocking every other request for as long as they take to load, which is 30-60 seconds.  The problem is not present for unauthenticated users.

Is there a thumbnail cache of some sort that is bypassed for auth users?  Can I force it, from islandora or server-side?

[Peter Murray]

Wow — that is weird. Can you try narrowing down where the problem might be? I can think of a few things to try:

1. Try getting the thumbnail directly from the browser. This should be equivalent to the web page loading it, but you never know.

2. Try getting the thumbnail through the Fedora server. For instance, an Islandora thumbnail image with the URL http://..../islandora/object/islandora%3A33/datastream/TN/view would (likely [1]) have the corresponding Fedora URL of http://localhost:18080/fedora/objects/islandora:33/datastreams/TN

The images (and other object accesses) are drawn from the Fedora server through the Drupal/Islandora layer so the access permissions can be applied. Trying to get to it directly from the Fedora server might narrow down where the issue is.

[1] “likely” means a default setup and having the Fedora server open to the world. Your mileage may vary.



Peter

On Dec 2, 2013, at 2:15 PM, <aaron...@commonmediainc.com> <aaron...@commonmediainc.com> wrote:

> Here is a screenshot of my network tab in chrome dev tools:
>
>
>

> --
> You received this message because you are subscribed to the Google Groups "islandora" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to islandora+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

--
Peter Murray
Assistant Director, Technology Services Development
LYRASIS
Peter....@lyrasis.org
+1 678-235-2955
800.999.8558 x2955

[aaron...@commonmediainc.com]

When I just access the image in my browser (islandora/object/islandora:1255/datastream/TN/view) i get the same response times as I do for in-situ images.

When I access the image through the fedora repo ( http://fedoraserver.com:8080/fedora/objects/islandora:1255/datastreams/TN) response time seems to be 'snappy', but when I click on "view content of this datastream" it downloads .bin file.  At least it does it quickly.

**edit**

We have http basic auth configured and after you submit your credentials, the server does take about 20 seconds to respond which it suspiciously close to our problem's lag.  Once authenticated it seems to be fine.  Could our setup be continuously re-authenticating with every thumbnail?

[Phil Redmon]

Check out:

https://groups.google.com/forum/#!searchin/islandora/thumbnail$20phil/islandora/nLNuzH2LtBs/h5INTUmHcqEJ

Specifially

https://groups.google.com/d/msg/islandora/nLNuzH2LtBs/8Zc7QOsRcQoJ

see if that resolves anything.

phil

[Peter Murray]

On Dec 3, 2013, at 9:01 AM, <aaron...@commonmediainc.com> <aaron...@commonmediainc.com> wrote:

> When I just access the image in my browser (islandora/object/islandora:1255/datastream/TN/view) i get the same response times as I do for in-situ images.
>
> When I access the image through the fedora repo ( http://fedoraserver.com:8080/fedora/objects/islandora:1255/datastreams/TN) response time seems to be 'snappy', but when I click on "view content of this datastream" it downloads .bin file. At least it does it quickly.
>
> **edit**
>
> We have http basic auth configured and after you submit your credentials, the server does take about 20 seconds to respond which it suspiciously close to our problem's lag. Once authenticated it seems to be fine. Could our setup be continuously re-authenticating with every thumbnail?

Downloading-as-bin might be a symptom of a problem noted on the mailing list last month:

https://groups.google.com/d/msg/islandora/CqoKtfqkhwo/OUxhR5qjYHYJ

Is the HTTP basic auth on the Islandora/Drupal server or the Fedora server? I couldn’t quite tell from the context of your message. In any case, I wouldn’t think there is much overhead because the whichever server is doing it would be doing it for every web request. There might be an overhead of an added HTTP round-trip if the client isn’t sending the “Authorization” header with the first request and the server has to respond with a 401 message.


Peter

[aaron...@commonmediainc.com]

The basic auth is on the fedora server, so prompt occurs at http://<fedora server>:8080/fedora/objects/islandora:1255/datastreams/TN/.

It looks like in this case, the delay also occurs with /DC and /TECHMD and supposedly all the other datastreams.  Perhaps I am not noticing those take so long when I load a page because when Drupal renders the page, it will waith to return anything whereas <img> tags are returned immediately and those datastreams load after the page HTML is returned.

Once I authenticate with the Basic Auth, speed seems normal.

I made the JAVA_OPT and djakota cache size edits and they do not seem to have made a difference.

Another strange thing I just noticed: if I try to access (all directly from fedora) /fedora/objects/islandora:1255/datastreams/TN/content it immediately downloads the .bin, whereas if I visit /fedora/objects/islandora:1255/datastreams/TN (substract /content) then I get the basic auth and the 20-30 second delay.

[aaron...@commonmediainc.com]

So, Wow... just fixed this issue and was it ever a stinker.

In order to work on this installation, I set up a local installation of fedora, and I had to open my local drupal database to the filter based on my IP address from the fedora server over the VPN.

And of course, that IP changed.

So every time an authenticated request came through, Fedora looks like it checks every entry in the Drupal Filter.  It hit the entry for the live installation and works just fine, and then it tried to hit my database, times out (after 20-30 seconds or whatever) and then continued on its merry way to serve my page, but without giving any indication that the Drupal Filter had failed to connect to a database.

Several of those timed-out database requests in a row turn out to be entirely the reason for the slow performance.  After updating my IP in the drupal filter, speed is basically comparable to Anonymous traffic.

[Peter Murray]

Huh, definitely ‘wow’. I hadn’t speculated that the filter worked that way. I would think that as soon as it got a valid user that it would skip the rest of the database checks. This might be ticket-worthy.


Peter

On Dec 3, 2013, at 2:17 PM, <aaron...@commonmediainc.com> <aaron...@commonmediainc.com> wrote:

> So, Wow... just fixed this issue and was it ever a stinker.
>
> In order to work on this installation, I set up a local installation of fedora, and I had to open my local drupal database to the filter based on my IP address from the fedora server over the VPN.
>
> And of course, that IP changed.
>
> So every time an authenticated request came through, Fedora looks like it checks every entry in the Drupal Filter. It hit the entry for the live installation and works just fine, and then it tried to hit my database, times out (after 20-30 seconds or whatever) and then continued on its merry way to serve my page, but without giving any indication that the Drupal Filter had failed to connect to a database.
>
> Several of those timed-out database requests in a row turn out to be entirely the reason for the slow performance. After updating my IP in the drupal filter, speed is basically comparable to Anonymous traffic.
>

[aaron...@commonmediainc.com]

Where can I file the bug?

[Peter Murray]

Do you have a login on DuraSpace’s Jira instance?

https://jira.duraspace.org/browse/ISLANDORA

That’d be the place to put it.


Peter

> --
> You received this message because you are subscribed to the Google Groups "islandora" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to islandora+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[aaron...@commonmediainc.com]

Filed!

Thanks to everyone who took to time to reply!

[Melissa Anez]

Thanks, Aaron! We'll take up the ticket for discussion at the Committers Call next week.

- Melissa