BonCodeAJP Connector - Generic Connector Communication Error

1,104 views
Skip to first unread message

Jason Tully

unread,
Apr 14, 2015, 5:07:53 PM4/14/15
to lu...@googlegroups.com

So I finally bit the bullet and updated our production server to Lucee. We updated our test server to Lucee a few weeks ago and haven’t had any issue. The software between our test and production environment is identical. The only difference is the volume of traffic. Before upgrading from Railo to Lucee, our install was rock solid. We could go months without a single reboot of Tomcat/Railo or IIS. However, that’s all changed since upgrading to Lucee.  After we upgraded, we made sure all of our settings from our server.xml were merged with Lucee’s install of the new Server.xml. TomCat doesn't seem to be crashing as we have FusionReactor running in the background and we can still serve up requests to any website via port 8888.

I believe the core of the issue is with the BonCode connector.  Sporadically we receive the following message.

Generic Connector Communication Error:

Please check and adjust your setup:

Ensure that Tomcat is running on given host and port.

If this is a timeout error consider adjusting IIS timeout by changing executionTimeout attribute in web.config (see manual).

 

An immediate refresh of the page/website and it loads right up. There is no way to reproduce the error. It occurs on just about all of our roughly 85 sites randomly throughout the day. Prior to upgrading to Lucee we never had this issue. In case its asked, all of our IIS application pools have the IIS default 20min timeout. Some higher traffic sites are set to "AlwaysRunning"


We enabled logging on the boncode connector, and see this in just about all of the boncode logs:

2015-04-14 07:27:15 1.0.21 ERROR

TCP Client level -- Server/Port:localhost/8009

Unable to write data to the transport connection: An established connection was aborted by the software in your host machine.

2015-04-14 07:27:15 One Connection raised an error

 

I’ve included some settings below that I believe are relevant to diagnosing the issue.

BonCode Version:

BonCodeAJP Connector Version 1.0.21
using setting file in C:\Windows\BonCodeAJP13.settings

Lucee/Server Info

Windows Server 2012 R1
IIS 8
Lucee 4.5.1.000 final
Apache Tomcat/7.0.61  <-- I tried several versions of tomcat
Java 1.7.0_76 (Oracle Corporation) 64bit 

4096mb Ram dedicated to Lucee (32gigs server total)


BonCodeAJP13.settings

<Settings>
<Server>localhost</Server>
<Port>8009</Port>
<MaxConnections>0</MaxConnections>
<EnableRemoteAdmin>True</EnableRemoteAdmin>
<EnableHeaderDataSupport>True</EnableHeaderDataSupport>
<ForceSecureSession>False</ForceSecureSession>
<AllowEmptyHeaders>False</AllowEmptyHeaders>
<LogLevel>3</LogLevel>
<LogDir>C:\Lucee\BonCodeLogs</LogDir>
</Settings> 


A note about the above: I had <MaxConnections> set equal to what I had in the server.xml, but that didn't help with the error.


Connector settings in TomCat Server.xml

<Connector port="8009"
    acceptorThreadCount="6"
    maxThreads="1500"
    minSpareThreads="150"
    maxSpareThreads="500"  
    tcpNoDelay="true"
    connectionTimeout="180000"
    enableLookups="false"
    protocol="AJP/1.3"
    redirectPort="8443"
/> 


A note about the above: In our Railo install, we never had a connectionTImeout attribute, but figured I would see if it would help.


Would appreciate any help anyone can offer.

Jordan Michaels

unread,
Apr 14, 2015, 5:36:25 PM4/14/15
to lu...@googlegroups.com
I would recommend turing on logging within BonCode to get more detail on
what is actually happening with the communication between IIS and Tomcat.

You can turn on logging by adding/editing the LogLevel, LogDir, and
LogFile params in the BonCode Connector config:
http://boncode.net/connector/webdocs/Tomcat_Connector.htm#_Toc415060639

Warm Regards,
Jordan Michaels

On 04/14/2015 02:07 PM, Jason Tully wrote:
> So I finally bit the bullet and updated our production server to Lucee.
> We updated our test server to Lucee a few weeks ago and haven’t had any
> issue. The software between our test and production environment is
> identical. The only difference is the volume of traffic. Before
> upgrading from Railo to Lucee, our install was rock solid. We could go
> months without a single reboot of Tomcat/Railo or IIS. However, that’s
> all changed since upgrading to Lucee. After we upgraded, we made sure
> all of our settings from our server.xml were merged with Lucee’s install
> of the new Server.xml. TomCat doesn't seem to be crashing as we have
> FusionReactor running in the background and we can still serve up
> requests to any website via port 8888.
>
> I believe the core of the issue is with the BonCode connector.
> Sporadically we receive the following message.
>
> *Generic Connector Communication Error:*
>
> *Please check and adjust your setup:*
>
> *Ensure that Tomcat is running on given host and port.*
>
> *If this is a timeout error consider adjusting IIS timeout by changing
> executionTimeout attribute in web.config (see manual).*
>
>
> An immediate refresh of the page/website and it loads right up. There is
> no way to reproduce the error. It occurs on just about all of our
> roughly 85 sites randomly throughout the day. Prior to upgrading to
> Lucee we never had this issue. In case its asked, all of our IIS
> application pools have the IIS default 20min timeout. Some higher
> traffic sites are set to "AlwaysRunning"
>
>
> We enabled logging on the boncode connector, and see this in just about
> all of the boncode logs:
>
> *2015-04-14 07:27:15 1.0.21 ERROR *
>
> *TCP Client level -- Server/Port:localhost/8009*
>
> *Unable to write data to the transport connection: An established
> connection was aborted by the software in your host machine.*
>
> *2015-04-14 07:27:15 One Connection raised an error*
>
> I’ve included some settings below that I believe are relevant to
> diagnosing the issue.
>
> *BonCode Version:*
>
> |
> BonCodeAJPConnectorVersion1.0.21
> usingsetting file inC:\Windows\BonCodeAJP13.settings
> |
>
> *Lucee/Server Info*
>
> Windows Server 2012 R1
> IIS 8
> Lucee 4.5.1.000 final
> Apache Tomcat/7.0.61 <-- I tried several versions of tomcat
> Java 1.7.0_76 (Oracle Corporation) 64bit
>
> 4096mb Ram dedicated to Lucee (32gigs server total)
>
>
> *BonCodeAJP13.settings*
>
> |
> <Settings>
> <Server>localhost</Server>
> <Port>8009</Port>
> <MaxConnections>0</MaxConnections>
> <EnableRemoteAdmin>True</EnableRemoteAdmin>
> <EnableHeaderDataSupport>True</EnableHeaderDataSupport>
> <ForceSecureSession>False</ForceSecureSession>
> <AllowEmptyHeaders>False</AllowEmptyHeaders>
> <LogLevel>3</LogLevel>
> <LogDir>C:\Lucee\BonCodeLogs</LogDir>
> </Settings>
>
> |
>
>
> A note about the above: I had <MaxConnections> set equal to what I had
> in the server.xml, but that didn't help with the error.
>
>
> *Connector settings in TomCat Server.xml*
>
> |
> <Connector port="8009"
> acceptorThreadCount="6"
> maxThreads="1500"
> minSpareThreads="150"
> maxSpareThreads="500"
> tcpNoDelay="true"
> connectionTimeout="180000"
> enableLookups="false"
> protocol="AJP/1.3"
> redirectPort="8443"
> />
> |
>
> A note about the above: In our Railo install, we never had a
> connectionTImeout attribute, but figured I would see if it would help.
>
>
> Would appreciate any help anyone can offer.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Lucee" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to lucee+un...@googlegroups.com
> <mailto:lucee+un...@googlegroups.com>.
> To post to this group, send email to lu...@googlegroups.com
> <mailto:lu...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/lucee/63e07ce1-9bd4-410e-b672-f2843253dad1%40googlegroups.com
> <https://groups.google.com/d/msgid/lucee/63e07ce1-9bd4-410e-b672-f2843253dad1%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

Jason Tully

unread,
Apr 14, 2015, 5:59:13 PM4/14/15
to lu...@googlegroups.com
Hi Jordan, Thanks for the reply. I already had logging enabled. The message from the log is in my original post. 

Jordan Michaels

unread,
Apr 14, 2015, 6:05:41 PM4/14/15
to lu...@googlegroups.com
::facepalm:: My apologies, and thanks for your patient reply Jason.

In this case, I would check the Tomcat catalina.out log file to see if
you're hitting any connection limits or something of that sort. Tomcat
is usually pretty good about logging issues. If it's closing connections
prematurely or hitting a limit of some kind, it will usually tell you.
That will give you some direction as to what connection settings to
adjust within your server.xml config.

Again, thanks for your patient response. That was very considerate of
you. ;)

Warm Regards,
Jordan Michaels
> > an email to lucee+un...@googlegroups.com <javascript:>
> > <mailto:lucee+un...@googlegroups.com <javascript:>>.
> > To post to this group, send email to lu...@googlegroups.com
> <javascript:>
> > <mailto:lu...@googlegroups.com <javascript:>>.
> <https://groups.google.com/d/msgid/lucee/63e07ce1-9bd4-410e-b672-f2843253dad1%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Lucee" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to lucee+un...@googlegroups.com
> <mailto:lucee+un...@googlegroups.com>.
> To post to this group, send email to lu...@googlegroups.com
> <mailto:lu...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/lucee/b6a19568-3936-49d6-86cb-da9b0f6c9303%40googlegroups.com
> <https://groups.google.com/d/msgid/lucee/b6a19568-3936-49d6-86cb-da9b0f6c9303%40googlegroups.com?utm_medium=email&utm_source=footer>.

Jason Tully

unread,
Apr 14, 2015, 9:27:39 PM4/14/15
to lu...@googlegroups.com
Spent the night combing through catalinia.*.log, and didn't find any errors. It was just full of a ton of "deployDirectory" logs. Also checked the other logs, and nothing unusual found. Any other suggestions?

Bilal

unread,
Apr 15, 2015, 11:29:21 AM4/15/15
to lu...@googlegroups.com
Jason,
to isolate the source of the issue
can you temporarily try to force re-connections on the connector side and let us know if this does it for you:


In the BonCodeAJP13.settings (normally in c:\windows) file, add or change a line like this:\
<MaxConnections>0</MaxConnections>

Then, restart IIS.

Thanks,
Bilal


Jason Tully

unread,
Apr 17, 2015, 7:08:53 AM4/17/15
to lu...@googlegroups.com
Bilal, Thank you. I have the that setting enabled (<MaxConnections>0</MaxConnections>) and from what I can see, the problem has gone away. Strange that it worked on Railo for quite sometime without this setting.

Is there any downside/trade-off to having it re-establish a connection every time using this setting?  

Thanks

Bilal

unread,
Apr 18, 2015, 12:28:27 PM4/18/15
to lu...@googlegroups.com
Investigating the issue for now. 

There is a slight overhead for establishing connections but if your CPU is not going critical you will be OK keeping the setting as is. 
Reply all
Reply to author
Forward
0 new messages