Issues with Apache - Tomcat Proxy

[Randy Johnson]

I am having an issue with Apache - Tomcat running railo.

The websites will not respond unless I restart apache.


Here is my apache conf entry:

ProxyPreserveHost On
ProxyPass / ajp://localhost:8009/
ProxyPassReverse / ajp://localhost:8009/

RewriteEngine On
ProxyRequests Off
RewriteCond %{REQUEST_URI} !^.*/(fancybox|flex2gateway|jrunscripts|
cfide|cfformgateway|railo-context|files|images|javascripts|
miscellaneous|stylesheets|robots.txt|sitemap.xml|rewrite.cfm|index.cfm|
favicon.ico|ckfinder|ckeditor|assets|product-images|store)($|/.*$)
[NC]

RewriteRule "^/(.*)" ajp://localhost:8009/rewrite.cfm/$1 [P,QSA,L]

Here is the Tomcat errors I am receiving:

Aug 25, 2011 10:32:32 AM org.apache.jk.common.ChannelSocket
processConnection
WARNING: processCallbacks status 2
Aug 25, 2011 10:32:32 AM org.apache.jk.common.ChannelSocket
processConnection
WARNING: processCallbacks status 2
Aug 25, 2011 10:32:32 AM org.apache.jk.core.MsgContext action
WARNING: Unable to send headers
java.net.SocketException: Broken pipe



Here is the apache errors I am receiving:

[Mon Sep 05 21:33:42 2011] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header
[Mon Sep 05 21:36:32 2011] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header
[Mon Sep 05 21:37:07 2011] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header
[Mon Sep 05 21:37:20 2011] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header
[Mon Sep 05 21:37:23 2011] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header
[Mon Sep 05 21:53:32 2011] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header
[Mon Sep 05 21:54:41 2011] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header
[Mon Sep 05 22:04:18 2011] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header
[Mon Sep 05 22:16:20 2011] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header


Has anyone seen this before? I have googled and tried a few things
but nothing seems to work.

Thanks!

Randy

[AJ Mercer]

this may not fix this issue, but it is something you will have to do

in tomact\conf\web.xml

add

<!-- CFWheels -->

<servlet-mapping>

    <servlet-name>GlobalCFMLServlet</servlet-name>

    <url-pattern>/rewrite.cfm/*</url-pattern>

</servlet-mapping>

--

AJ Mercer
<webonix:net strength="Industrial" /> | <webonix:org community="Open" />
http://twitter.com/webonix

Railo Community Manager

[Randy Johnson]

AJ,

Yep you are right. I do have that. I should also specify, that my
setup works normal for a few hours and then out of the blue the sites
stop responding with the errors in each of the logs like I mentioned
above.

Thanks,
Randy

On Sep 5, 8:19 pm, AJ Mercer <ajmer...@gmail.com> wrote:
> this may not fix this issue, but it is something you will have to do
> in tomact\conf\web.xml
> add
>
> <!-- CFWheels -->
> <servlet-mapping>
>     <servlet-name>GlobalCFMLServlet</servlet-name>
>     <url-pattern>/rewrite.cfm/*</url-pattern>
> </servlet-mapping>
>

> *AJ Mercer*
> <webonix:net strength="Industrial" /> <http://webonix.net> | <webonix:org
> community="Open" /> <http://webonix.org>http://twitter.com/webonix
> Railo Community Manager <http://www.getrailo.org/index.cfm/community/team/>

[Justin Edwards]

Randy,

This will happen if railo runs out of memory, and is continually
garbage collecting. It may also happen if railo has continually high
CPU usage. It is basically Denial of Serviced with requests that it
can't handle, and the connector will throw errors because it can't
communicate.

[Randy Johnson]

It makes sense what you are saying.

Why is it when I this happens I reset apache and everything is fine?
Is the couple of seconds it takes apache to reset enough for tomcat to
recover?

Randy

On Sep 5, 9:47 pm, Justin Edwards <justinledwa...@gmail.com> wrote:
> Randy,
>
> This will happen if railo runs out of memory, and is continually
> garbage collecting.   It may also happen if railo has continually high
> CPU usage.  It is basically Denial of Serviced with requests that it
> can't handle, and the connector will throw errors because it can't
> communicate.
>

[Justin Edwards]

I would say no. What is your tomcat memory limit set as?

[Randy Johnson]

JAVA_OPTS = "-Xms256m -Xmx512m -XX:MaxPermSize=128m "; # memory
settings

On Sep 6, 10:06 am, Justin Edwards <justinledwa...@gmail.com> wrote:
> I would say no.  What is your tomcat memory limit set as?
>

[Justin Edwards]

I would bump up the xmx to whatever is reasonable for your server.
If you can, keep an eye on how much memory it is using. If you
notices that it peaks often and goes back down, you may need to do
some limiting on queries or something related.

I've come across a lot of code that limits on the viewing end, but
doesn't have a TOP or LIMIT on the queries.

[whostheJBoss]

I've noticed this happen when I restart Tomcat while Apache is still running, even after Tomcat has restarted, I get 500 until I restart Apache.

[Todd Rafferty]

Are you using mod_proxy or Jakarta? I never get this with mod_proxy.

On Wed, Sep 7, 2011 at 10:25 AM, whostheJBoss <j...@grapestack.com> wrote:

I've noticed this happen when I restart Tomcat while Apache is still running, even after Tomcat has restarted, I get 500 until I restart Apache.

--
~Todd Rafferty
Volunteer Community Manager
Railo Server - Open Source
----

Get Railo: http://getrailo.org/
Write Documentation: http://wiki.getrailo.org/

Report Bugs: https://issues.jboss.org/browse/RAILO
Request Features: https://railo.uservoice.com/

[whostheJBoss]

AJP Proxy Balancer...

<Proxy balancer://mycluster>

        BalancerMember ajp://tomcat1:8009/ route=jvm1 loadfactor=50

        BalancerMember ajp://tomcat2:8009/ route=jvm2 loadfactor=50

        ProxySet lbmethod=byrequests

</Proxy>

[Todd Rafferty]

Never messed with it. Very strange that Apache would choke or care.

2011/9/7 whostheJBoss <j...@grapestack.com>

[Justin Edwards]

Not sure if you have this already and didn't post it, but it looks
like you could need to define the stickiness.

http://httpd.apache.org/docs/2.1/mod/mod_proxy_balancer.html#stickyness_implementation

2011/9/7 whostheJBoss <j...@grapestack.com>:

[whostheJBoss]

It's set on my server:

         ProxySet stickysession=JSESSIONID

And the sticky sessions work fine. Like I said, it's just a weird thing that happens if I restart Tomcat *sometimes* and then try to access the page before restarting Apache. What's weird is that I can go to :8080 and see the site, but Apache on 80 still gives me 500 until I restart it.

This happens when I just have one Apache and one Tomcat server, no load balancing. It happens on my windows box sometimes too, I'll restart Tomcat, but I can't hit it through 80 until I restart Apache. It's rare, but it happens.

[Justin Edwards]

I think I may have had this problem before on our dev servers, but
it's definitely mod proxy ajp related. I've moved our dev servers
over to cherokee, and our dev servers are on 10.04 ubuntu server. I
remember having to reset apache httpd and tomcat.

For some reason lately the most problems I've had are browser related,
they don't clear the cache properly after I've told them to and I keep
refreshing, then I'll close out the tab and go to the site again and
it will be fine. This is on chrome and firefox.