issue with running tomcat behind apache where apache throws periodic 503 errors

[Tyler Danstrom]

Hi,

We are having an issue with our DSpace install where Tomcat appears to idle out during any period of sustained inactivity and then Apache throws a 503 error the next time any user attempts to connect to our dspace instance.

Dspace version: 5.1

UI type: xmlui

OS: Redhat Enterprise Linux release 6.7

database: oracle

Tomcat version: 7.0.59

Apache version:2.2.15

Tomcat config:

<Connector port="8009" protocol="AJP/1.3" redirectPort="8443"

URIEncoding="UTF-8" connectionTimeout="20000"/>

 

Apache config:

ProxyPass /jspui ajp://localhost:8009/jspui/ timeout=10 retry=0

ProxyPass /xmlui ajp://localhost:8009/xmlui/ timeout=10 retry=0

We are running Tomcat behind Apache with Shibboleth and accessing dspace with the following path:

/xmlui 

after the host name

The connection is not encrypted over ssl by default.

Narrative of the problem

Any time that there is a period of inactivity (inactivity is defined as no traffic) Tomcat appears to cease to respond to Apache and throws a 503 error to the end-user.

The solution to this problem seems to be to upon first receiving the 503 error immediately reload the page (this may only be required once but has been witnessed to require three times) within seconds of the unsuccessful attempt.

Eventually, the xmlui url resolves correctly and returns a successful connection to the dspace application and from then on until there is another sustand period of inactivity everything works fine for one or multiple simultaneous connections.

Has anyone else experienced this issue? Does anyone know of a long-term solution to this problem?

Any and all help is immensely appreciated.

Sincerely,

Tyler Danstrom

Programmer/Analyst

University of Chicago Library

[helix84]

Never experienced this problem. 503 only ever correctly appears only when tomcat is down. Here's my configuration:

Apache 2.2, 2.4:

        ProxyPass / ajp://localhost:8009/ retry=2

        ProxyPassReverse / ajp://localhost:8009/

Tomcat 7, 8:

    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" />

I probably wouldn't set retry to 0. Try to turn on the keepalive parameter:

http://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypass

Regards,
~~helix84

Compulsory reading: DSpace Mailing List Etiquette
https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

[Tyler Danstrom]

Hi,

So, after more investigation we've figured out that the problem is with the ajp ilink

The following appears in our logs

[Wed Sep 16 14:51:35 2015] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header

[Wed Sep 16 14:51:35 2015] [error] ajp_read_header: ajp_ilink_receive failed

[Wed Sep 16 14:51:35 2015] [error] (70007)The timeout specified has expired: proxy: read response failed from (null) (localhost)

 

Does anyone know what DSpace recommends to set on timeout with an AJP link?

[Pottinger, Hardy J.]

Hi, we recently encountered a similar situation, and in researching default timeouts, I discovered the default timeout for AJP sessions is 300 seconds. Not knowing exactly *why* our sessions need to last a bit longer, I decided to try a really big number in that timeout:

# let's let proxy connections live a whole 24 hours (i.e. 86400 seconds) the default is 300 seconds
ProxyTimeout 86400

So far, I haven't heard any more complaints about 503 errors.

Now, you may not be comfortable using such a large number for your timeout. I'd urge caution setting this number. If anyone has a suggestion for a better/more-reasonable number for the AJP timeout, I'd love to hear it.

Thanks!

--Hardy

---

From: dspac...@googlegroups.com [dspac...@googlegroups.com] on behalf of Tyler Danstrom [ty...@danstrom.com]
Sent: Tuesday, September 22, 2015 9:38 AM
To: DSpace Technical Support
Cc: ty...@danstrom.com; hel...@centrum.sk
Subject: Re: [dspace-tech] issue with running tomcat behind apache where apache throws periodic 503 errors

--
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech...@googlegroups.com.
To post to this group, send email to dspac...@googlegroups.com.
Visit this group at http://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.

[Tim Donohue]

Hi Tyler,

In case it's of any help..

For sites I've managed in the past, I've always had the following
general settings in Apache (with mod_proxy and mod_proxy_ajp):

## Proxy / Forwarding Settings ##
<Proxy *>
AddDefaultCharset Off
Order deny,allow
Allow from all
</Proxy>

# Pass all requests to Tomcat's AJP Connector

ProxyPass / ajp://localhost:8009/

ProxyPassReverse / ajp://localhost:8009/

In Tomcat 7, I just have:

<Connector port="8009" protocol="AJP/1.3" URIEncoding="UTF-8"/>

I also setup the / (ROOT) path to just load my DSpace UI of choice.

Since you are using Shibboleth, you may also wish to review your
settings for mod_shib. I doubt it's the root issue, but it's possible
that Shibboleth could be a factor as well. We do have recommended
mod_shib settings documented at:
https://wiki.duraspace.org/display/DSDOC5x/Authentication+Plugins#AuthenticationPlugins-ShibbolethAuthentication

This StackOverflow answer regarding the same error message also points
to using either "timeout" (in ProxyPass) or ProxyTimeout (which Hardy
recommended):
http://serverfault.com/questions/655362/mod-proxy-ajp-70007the-timeout-specified-has-expired-ajp-ilink-receive-can

Good luck! Let us know if you find a resolution, so that we can add it
to the wiki/documentation if needed.

- Tim

> <https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette>
>
> --
> You received this message because you are subscribed to the Google
> Groups "DSpace Technical Support" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to dspace-tech...@googlegroups.com

> <mailto:dspace-tech...@googlegroups.com>.

> To post to this group, send email to dspac...@googlegroups.com

> <mailto:dspac...@googlegroups.com>.

> Visit this group at http://groups.google.com/group/dspace-tech.
> For more options, visit https://groups.google.com/d/optout.

--
Tim Donohue
Technical Lead for DSpace & DSpaceDirect
DuraSpace.org | DSpace.org | DSpaceDirect.org

[Graham Triggs]

Hi,

There are a couple of distinct issues that could be going on:

1) The Tomcat connector may be accepting fewer connections than Apache - so Apache requests have to wait for Tomcat to finish processing some requests.

Note - be careful in looking at Tomcat being "idle" - a lack of processor activity is not the same as the threads being idle. They may be sleeping, but allocated to a request.

2) You may be running out of permitted open ports, so Apache can't open another network connection until the OS times out some of the connections, even if Tomcat isn't doing anything.

G