Boncode/AJP Connector hanging, IIS 8.5, Railo 4.1.2

[Ed Barber]

The past two days we've been having an issue with our Railo/IIS setup.  2 or 3 times a day, requests to any coldfusion pages will suddenly hang. There won't be any errors returned, just simply the request never completes. This only happens with coldfusion pages (.cfm, .cfc).  We have a static HTML site that will still respond just fine. If we restart IIS, everything works again. Also, we can continue to access the Railo Start page by going to http://127.0.0.1:8888 on the server, even while requests routed through IIS hang. This leads me to believe it's an issue with the BonCode Connector.   I haven't been able to reproduce the issue on demand yet, I've tried using jMeter to hit the server with a ton of requets, but that doesn't seem to cause this issue to resurface. 

I've looked in all the logs I can think of, and nothing really jumps out. Wen I restart IIS, the event logs fills up with ASP.NET 3001 errors: "The request has been aborted". But nothing shows up in the event log while the issue is actually happening.

In the catalina.{date}.log, located in {railo-install}/tomcat/logs, there's the following

Jun 20, 2014 12:58:27 PM org.apache.coyote.ajp.AjpMessage processHeader
SEVERE: Invalid message received with signature 18245

But I believe that is showing up after I restart IIS. 

I'm running out of places to look. Any help would be greatly appreciated!

-Ed

[Ed Barber]

As an update, I just looked at the thread dump I took at the last occurrence, and found that the ajp-bio-8009-Acceptor thread is in a WAITING state...

"ajp-bio-8009-Acceptor-0" daemon prio=6 tid=0x0000000010fc8800 nid=0x3424 waiting on condition [0x0000000010d2f000]
   java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  <0x00000007801931f8> (a org.apache.tomcat.util.threads.LimitLatch$Sync)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
 at org.apache.tomcat.util.threads.LimitLatch.countUpOrAwait(LimitLatch.java:115)
 at org.apache.tomcat.util.net.AbstractEndpoint.countUpOrAwaitConnection(AbstractEndpoint.java:720)
 at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:210)
 at java.lang.Thread.run(Thread.java:744)

When everything is running fine, this thread is in a RUNNABLE state

[Jordan Michaels]

Hi Ed,

That's pretty unusual. I would suggest enabling logging from within the
BonCode connector and see if it's logs provide any clues. You can enable
logging in the BonCode connector by editing the settings file -
documentation is here:

http://boncode.net/connector/webdocs/Tomcat_Connector.htm#_Toc379210944

Specifically, Log Level, LogDir, LogFile.

Warm Regards,
Jordan Michaels

On 06/20/2014 10:15 AM, Ed Barber wrote:
> As an update, I just looked at the thread dump I took at the last
> occurrence, and found that the ajp-bio-8009-Acceptor thread is in a
> WAITING state...
> |
> "ajp-bio-8009-Acceptor-0"daemon

> prio=6tid=0x0000000010fc8800nid=0x3424waiting on condition

> [0x0000000010d2f000]
> java.lang.Thread.State:WAITING (parking)
> at sun.misc.Unsafe.park(NativeMethod)

> -parking to wait for<0x00000007801931f8>(a

> <http://ASP.NET> 3001 errors: "The request has been aborted". But

> nothing shows up in the event log while the issue is actually happening.
>
> In the catalina.{date}.log, located in {railo-install}/tomcat/logs,
> there's the following
> |
> Jun20,201412:58:27PM org.apache.coyote.ajp.AjpMessageprocessHeader

> SEVERE:Invalidmessage received withsignature 18245

> |
>
> But I believe that is showing up after I restart IIS.
>
> I'm running out of places to look. Any help would be greatly
> appreciated!
>
> -Ed
>

> --
> Did you find this reply useful? Help the Railo community and add it to
> the Railo Server wiki at https://github.com/getrailo/railo/wiki
> ---
> You received this message because you are subscribed to the Google
> Groups "Railo" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/railo/a3835b6c-6d80-411e-bef4-257838e2fbda%40googlegroups.com
> <https://groups.google.com/d/msgid/railo/a3835b6c-6d80-411e-bef4-257838e2fbda%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Ed Barber]

Thanks Jordan. After much investigation, I determined it was an issue with BonCode not closing connections properly. So new connections would keep opening, but would never close. Once it hit the max connection limit, (default 200), it stopped accepting new connections. I'm using a slightly outdated version, 1.0.16 (current is 1.0.19), but the solution was to add a connectionTimeout property to the Connector node in server.xml (found in {railo-install}/tomcat/conf/server.xml.

The default looked like 

<Connector port="8009" protocol="AJP/1.3" redirectPort="8443"/>

I changed it to 

<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" connectionTimeout="300000" keepAliveTimeout="300000"/>

I changed this, restarted Railo, and then connections began closing properly. Haven't had an issue since.

-Ed

[Jordan Michaels]

Awesome. Thanks for following up with the solution Ed. The google
archives and I appreciate it! =)

Warm Regards,
Jordan Michaels

On 06/26/2014 07:31 AM, Ed Barber wrote:
> Thanks Jordan. After much investigation, I determined it was an issue
> with BonCode not closing connections properly. So new connections would
> keep opening, but would never close. Once it hit the max connection
> limit, (default 200), it stopped accepting new connections. I'm using a
> slightly outdated version, 1.0.16 (current is 1.0.19), but the solution
> was to add a connectionTimeout property to the Connector node in
> server.xml (found in {railo-install}/tomcat/conf/server.xml.
>
> The default looked like
> |

> <Connectorport="8009"protocol="AJP/1.3"redirectPort="8443"/>
> |
>
> I changed it to
> |
> <Connectorport="8009"protocol="AJP/1.3"redirectPort="8443"connectionTimeout="300000"keepAliveTimeout="300000"/>

> <https://groups.google.com/d/msgid/railo/a3835b6c-6d80-411e-bef4-257838e2fbda%40googlegroups.com?utm_medium=email&utm_source=footer

> <https://groups.google.com/d/msgid/railo/a3835b6c-6d80-411e-bef4-257838e2fbda%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
> --
> Did you find this reply useful? Help the Railo community and add it to
> the Railo Server wiki at https://github.com/getrailo/railo/wiki
> ---
> You received this message because you are subscribed to the Google
> Groups "Railo" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/railo/671bf89b-d025-49a1-95c2-f8ce6fe45dfb%40googlegroups.com
> <https://groups.google.com/d/msgid/railo/671bf89b-d025-49a1-95c2-f8ce6fe45dfb%40googlegroups.com?utm_medium=email&utm_source=footer>.