Tomcat refusing connections (or is it BonCode?)

[John Waddleton]

I know there are already posts on this mailing list and elsewhere on the web with questions similar to mine. I've been researching this for days and I thoroughly read all that I could find. But each case seems to be slightly different than mine, and I've tried most of the suggestions in the posts I've found without solving my problem. So, sorry i advance if I'm missing anything obvious!

I am running Railo 4.2.1 / Tomcat 7.0.47 on Windows 2008r2 (4GB) with IIS7.5 connecting via Boncode connector. The system supports just over 200 contexts for a total of 79 active websites in a production environment. My problem is that after an indeterminate time all the websites go down displaying the message: "No connection could be made because the target machine actively refused it. 127.0.0.1:8009". This condition persists until I restart Tomcat/Railo. In some cases, by reviewing the log files, I conclude that the condition has lasted up to 3-4 hours before I became aware of the problem and restarted. I have some evidence that in a few cases the problem may have self-corrected, but I can't confirm this assertion.

The problem now occurs only once every 2-3 days (originally once or twice a day), and not normally on the weekends during lower traffic periods. I've spent literally a few weeks trying to solve this problem by making adjustments and tweaks and then waiting for the sites to fail again. I seem to have increased the time between failures by increasing the maxThreads value in server.xml to 2500, but the problem persists. And, based on some other information I have found, I am concerned that 2500 may be too high for a four core, single processor server.

Reviewing my log files, I begin to think that my problem stems from an Out of Memory, Java heap space error. On the days when the system has failed, the log files are peppered with references to the Out of Memory condition prior to my restarting Railo/Tomcat. Could an OOM error lead to Tomcat refusing connections?

In addition, in most cases but not all, I note the following type of error in catalina.xxxx-xx-xx.log prior to my restarting Railo/Tomcat:

Jul 28, 2014 4:01:35 AM com.microsoft.sqlserver.jdbc.TDSParser throwUnexpectedTokenException

SEVERE: ConnectionID:10182 ClientConnectionId: d7ffa528-e0fc-4109-be37-df070132e53c: getNextResult: Encountered unexpected unknown token (0x4)

Jul 28, 2014 4:01:35 AM com.microsoft.sqlserver.jdbc.TDSReader throwInvalidTDSToken

SEVERE: ConnectionID:10182 ClientConnectionId: d7ffa528-e0fc-4109-be37-df070132e53c got unexpected value in TDS response at offset:6453

The offset and token values change with each error entry. I suspect these errors may be related to SQL 2K5 databases associated with some of our sites that include columns of data type "text", which I have learned is an invalid data type that is nonetheless supported by SQL2K5 and ACF MX without problem. I think, but I'm not sure, that changing the datatype to longvarchar may solve this? Input, anyone? Could these errors be leading to the OOM errors I'm seeing?

The Memory Usage Heap indicator in Railo administrator seems to run consistently between 80 and 95%, which seems high to me. Further, when I hover over the indicator, the total memory available seems to be set at 256M. Maybe the problem? I've tried increasing the heap size by creating setenv.bat in the tomcat/bin folder and populating it as follows:

set "JAVA_OPTS=-server -Xms512m -Xmx1536m -XX:MaxPermSize=768m"

But, upon restarting Railo/Tomcat, the percent used and total seem to remain the same as before. Am I missing something? My Next "experiment" will include:

set "CATALINA_OPTS=-server -Xms512m -Xmx1536m -XX:MaxPermSize=768m"

in the setenv.bat file...

While composing this message and reviewing everything I've been researching, I noticed that my MaxConnections setting in BonCodeAJP13.settings has until now been set at 200. The error message points to port 8009, so maybe BonCode is the culprit? I really have no idea how many concurrent connections we experience, and frankly I'm not sure how to check that statistic. However, according to the docs this setting is supposed to be the same as the MaxThreads value in server.xml, so my next "experiment" barring other input from this mailing list, will be to change this setting to 2500 also. Could this setting have been the issue all along?

I'm new to Railo, Tomcat, and the BonCode connector, so perhaps some of these questions have an obvious answer. But, I've been climbing a steep learning curve and I could use a little assistance!

The same sites ran flawlessly on a 6 year old Win2K/ACFMX standard install server with no problem and no special setup. I'm excited to be a new member of the Railo community, but it's been far more difficult to get everything set up and running than it was previously on ACF :( In the meantime, my customers aren't so happy with me. My fault, I guess. The first test sites I transferred worked flawlessly, so I jumped in with both feet and transferred everything. Now the system keeps crashing and my hours of research have left me feeling like I still don't get it!

HELP! and thanks in advance!

[Jordan Michaels]

Turn on logging in the boncode connector. That would tell you for sure
if it was boncode or a problem with boncode connecting to the Tomcat AJP
port. Once you know, then it's just a matter of settings.

Warm Regards,
Jordan Michaels

> --
> Did you find this reply useful? Help the Railo community and add it to
> the Railo Server wiki at https://github.com/getrailo/railo/wiki
> ---
> You received this message because you are subscribed to the Google
> Groups "Railo" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/railo/6f641cc9-c3c9-4a64-a7be-ad6e08d7481a%40googlegroups.com
> <https://groups.google.com/d/msgid/railo/6f641cc9-c3c9-4a64-a7be-ad6e08d7481a%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[John Waddleton]

Thanks for the suggestion. I created the c:\temp directory, and assigned read/write permissions for IIS_IUSRS group as directed in the BonCode User Manaul. I added the following lines to BonCodeAJP13.settings file in the AJP13 folder:

  <LogLevel>2</LogLevel>
  <LogDir>c:\temp</LogDir>

I then rebooted the server. Based on log level 2, shouldn't I expect to see a log created immediately upon rebooting? Unfortunately, I don't....and half an hour later I still don't

I'm beginning to feel really stupid about all this. Have I missed something simple in setting up the log that is preventing its creation? 

BTW, in the time since I originally posted my first request, the server has failed again, preceded in  the log file as before by both the java heap space error messages and unknown token/unknown value error messages, this time including a healthy dose of GC overhead limit exceeded messages. The error message I see when attempting to view our websites specifically states "Error connecting to Apache Tomcat", so it seems as if Tomcat has stopped listening.

Another related question - the railo-stderr log file entries all start out nicely dated and time stamped. At some point in the file this stops, and subsequent messages have no date/time information associated (including all the java heap space and GC overhead messages), making it hard to track exactly when everything is occurring. Is this expected behavior?

Thanks for continued help!

[Nando Breiter]

I am running Railo 4.2.1 / Tomcat 7.0.47 on Windows 2008r2 (4GB) with IIS7.5 connecting via Boncode connector. The system supports just over 200 contexts for a total of 79 active websites in a production environment.

Any possibility to easily increase the memory allocation here? The errors you are seeing indicate you are running out of memory. To my novice eye, it seems 4GB might be insufficient.

That said, the fact that the server crashes only after a time might point to a memory leak in your code. In this case, increasing memory allocation would increase the gap between crashes, and provide a more solid clue that you might be dealing with a memory leak.

Aria Media Sagl
Via Rompada 40
6987 Caslano
Switzerland

+41 (0)91 600 9601
+41 (0)76 303 4477 cell
skype: ariamedia

To view this discussion on the web visit https://groups.google.com/d/msgid/railo/f4f81c09-92da-48b6-92ac-4e3175d97857%40googlegroups.com.

[Nando Breiter]

The same sites ran flawlessly on a 6 year old Win2K/ACFMX standard install server

I've read somewhere in a thread about ColdFusion memory issues that earlier versions of operating systems and server software consumed less memory. If your setup worked comfortably with 4GB on Win2K and ACF 6(?), 4GB may not be sufficient with more recent releases ... again novice speculation.

Aria Media Sagl
Via Rompada 40
6987 Caslano
Switzerland

+41 (0)91 600 9601
+41 (0)76 303 4477 cell
skype: ariamedia

[John Waddleton]

Thanks for the input, Nando. I've considered increasing physical memory also, which we can do if necessary. However, I'm thinking it's a memory allocation issue rather than the physical memory. In the Railo administrator, under "Memory Usage" in the right side column, under the heading "Heap", my indicator is consistently in the 90% range, even immediately after restart. Hovering over that indicator at present reveals the following message:

3% available (6mb), 97% in use (247mb)

To me, this says only 256mb (253mb according to the math) has been allocated to the heap. Is this a correct interpretation of what I'm seeing? If so, I'm confused, because I have the following settings in setenv.bat:

set "JAVA_OPTS=-server -Xms512m -Xmx1536m -XX:MaxPermSize=768m"

set "CATALINA_OPTS=-server -Xms512m -Xmx1536m -XX:MaxPermSize=768m"

According to what I've been able to find online, this should allocate a minimum of 512mb to the heap. or, maybe being a novice to this myself, I'm misunderstanding something???

[Igal Sapir]

Are you running Tomcat as a Windows service?  If so then you set the memory values in the wrong place.  There is a Windows applet under Configure Tomcat (from the Start menu).  Use it, under the Java tab.

To view this discussion on the web visit https://groups.google.com/d/msgid/railo/ba8f7c05-08bd-4f02-8fd1-0a37faf42ddd%40googlegroups.com.

[John Waddleton]

I installed Tomcat as part of the Railo installation, so I don't really know the answer to your question. There is no entry for "Tomcat" in the services window and I don't see "Configure Tomcat" in my Windows start menu. To restart Tomcat, I restart the Railo service, so I'm thinking maybe the answer is no, it's not running as a service?

[Igal @ getRailo.org]

I'm not familiar with the settings that the installer makes, but if you're on Windows then you should find that applet.

maybe someone else (Jordan?) can point you to the correct name of that applet.

alternatively you can set it in the registry, but using the Configure utility will be much easier.

To view this discussion on the web visit https://groups.google.com/d/msgid/railo/e50070b5-941a-4983-81de-112581e32ee3%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

-- 
Igal Sapir
Railo Core Developer
http://getRailo.org/

[Bilal]

John,

there are a few things that we need to check on to make sure we are looking at the correct source of the issues.

First we need to isolate connector vs Tomcat issues:

a) When you receive these messages please do not immediatly restart your services, instead can you check what you receive when you directly access Tomcat without IIS.

You can do so by calling the same site (on server) on port 8888, e.g. http://mysite.com:8888/someurl/somepage.cfm

=> if above produces error or non-response your issue is most likely with Tomcat and not with connector. The connector message would, then, be correct. Tomcat is resource constraint and you should check, code, configuration etc.

b) If above produces a result. You can tweak the number of connection allowed by changing the MaxConnection (on BonCode setting file) and maxThreads in Server.xml of Tomcat:

e.g. <MaxConnections>1000</MaxConnections>  in BonCodeAJP13.settings file.

And in Setting.xml for tomcat:

<!-- Define an AJP 1.3 Connector on port 8009 -->

<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" maxThreads="1000" connectionTimeout="60000" />

In the above I am also including a timeout for idle connections on the Tomcat side so the connection threads can be released when they died unexpectedly.

HTH,

Bilal

[John Waddleton]

Thank you so much for your attention to this problem. 

I found something elsewhere on the web that makes me think the applet you are referring to is tomcat7w-win64.exe in the tomcat/bin directory. However, when I try to run that program, I receive the message "The specified service does not exist as an installed service. Unable to open the service 'tomcat7w-win64'. So maybe this confirms Tomcat is not running as a service?

Can you confirm for certain that what I'm seeing in the Railo administrator is showing the allocated java heap space, and that apparently the settings I've entered into setenv.bat are not being applied? If so, I'm really hoping this is the cause of my problems and that solving the memory issue will stop Tomcat from refusing connections.

...

[Igal @ getRailo.org]

you must add some arguments for it to work, e.g.:

    tomcat7w-win64.exe //ES//{Service-Name}

so if you are running a service named "Railo", it would be:

    tomcat7w-win64.exe //ES//Railo

but I'm sure that you have a shortcut to it from the Start menu somewhere...

--

Did you find this reply useful? Help the Railo community and add it to the Railo Server wiki at https://github.com/getrailo/railo/wiki
---
You received this message because you are subscribed to the Google Groups "Railo" group.

To view this discussion on the web visit https://groups.google.com/d/msgid/railo/aa3912aa-85a1-4f16-9c21-b20cb76b9e4a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[John Waddleton]

EXCELLENT! I added the arguments as suggested, ran it, and the Railo Server Properties window appeared as promised. I changed the Java memory settings, then restarted Railo and - TA-DAH - the administrator now shows the memory levels I expected! After running for about 20 minutes, the memory usage seems to  be stabilized around 20%, which is still over 300M and much higher than the previous max of 256M. I really, really hope this solves the problem!

I swear I couldn't find it on the start menu - but it's there now, because I added a shortcut myself!

I hope this thread will help others, as I've literally been searching for weeks for an answer to this, and while I found several references to the setenv.bat file, I found only one other reference to this applet, and that one wasn't clear in regards to the arguments necessary.

Just to help put my mind at ease on this, can you confirm for certain that OOM errors will cause Tomcat to actively refuse connections?

Again, thank you, thank you, thank you. This has been frustrating me for weeks and I have high hopes that this will solve the problem. (I feel stupid that it took so long to get here...)

...

[Igal @ getRailo.org]

OOME will cause the JVM to choke, and possibly  die, so "actively refuse connection" can definitely be a symptom of that.

--

Did you find this reply useful? Help the Railo community and add it to the Railo Server wiki at https://github.com/getrailo/railo/wiki
---
You received this message because you are subscribed to the Google Groups "Railo" group.

To view this discussion on the web visit https://groups.google.com/d/msgid/railo/954d6cbe-e154-4117-b120-aaf5a526539e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Alex Skinner]

Yes if you get out of memory errors tomcat will usually give up and stop servicing requests.

Then you see the error you were referring to

Sent from my phone

--

Did you find this reply useful? Help the Railo community and add it to the Railo Server wiki at https://github.com/getrailo/railo/wiki
---
You received this message because you are subscribed to the Google Groups "Railo" group.

To view this discussion on the web visit https://groups.google.com/d/msgid/railo/954d6cbe-e154-4117-b120-aaf5a526539e%40googlegroups.com.

[John Waddleton]

Bilal,

Thank you for your input. As you will no doubt see from my other post, Igal has directed me to the Railo Server Properties (which I probably stupidly overlook somewhere in the documentation), and I am now running at a much more comfortable memory usage level. I hope this will solve the problem!

In response to your suggestions: 

1. As of yesterday evening I had already increased the MaxConnections setting to 2500, which now matches the high MaxThreads setting I had previously adjusted about 2 weeks ago. Is there any downside to leaving the values that high?
   
2. I added the connectionTimeout setting per your suggestion, even though I have high hopes for the memory adjustment to be the solution. The connection timeout just seems to be a good idea anyway, right?

I hope I never have another connection refused by Tomcat, but if I do I will follow your suggestions regarding isolating Tomcat and the connector. As a point of clarification (which I'm in need of a lot lately), is it adequate just to go directly to http://mysite.com:8888 to hit Tomcat direct?

[Jordan Michaels]

It's in the "Java" tab of the Railo/Tomcat Service control. Here is a
good blog post with some screen shots:

http://blog.kukiel.net/2011/08/how-to-change-jvm-arguments-in-railo.html

Warm Regards,
Jordan Michaels

On 07/31/2014 11:15 AM, Igal @ getRailo.org wrote:
> I'm not familiar with the settings that the installer makes, but if
> you're on Windows then you should find that applet.
>
> maybe someone else (Jordan?) can point you to the correct name of that
> applet.
>
> alternatively you can set it in the registry, but using the Configure
> utility will be much easier.
>
>
> On 7/31/2014 11:11 AM, John Waddleton wrote:
>> I installed Tomcat as part of the Railo installation, so I don't
>> really know the answer to your question. There is no entry for
>> "Tomcat" in the services window and I don't see "Configure Tomcat" in
>> my Windows start menu. To restart Tomcat, I restart the Railo service,
>> so I'm thinking maybe the answer is no, it's not running as a service?
>>
>>
>> On Thursday, July 31, 2014 10:25:51 AM UTC-7, Igal wrote:
>>
>> Are you running Tomcat as a Windows service? If so then you set
>> the memory values in the wrong place. There is a Windows applet
>> under Configure Tomcat (from the Start menu). Use it, under the
>> Java tab.
>>
>> On Jul 31, 2014 9:59 AM, "John Waddleton" <jo...@waddleton.com

>> <javascript:>> wrote:
>>
>> Thanks for the input, Nando. I've considered increasing
>> physical memory also, which we can do if necessary. However,
>> I'm thinking it's a memory allocation issue rather than the
>> physical memory. In the Railo administrator, under "Memory
>> Usage" in the right side column, under the heading "Heap", my
>> indicator is consistently in the 90% range, even immediately
>> after restart. Hovering over that indicator at present reveals
>> the following message:
>>
>> 3% available (6mb), 97% in use (247mb)
>>
>> To me, this says only 256mb (253mb according to the math) has
>> been allocated to the heap. Is this a correct interpretation
>> of what I'm seeing? If so, I'm confused, because I have the
>> following settings in setenv.bat:
>>
>> set "JAVA_OPTS=-server -Xms512m -Xmx1536m -XX:MaxPermSize=768m"
>> set "CATALINA_OPTS=-server -Xms512m -Xmx1536m
>> -XX:MaxPermSize=768m"
>>
>> According to what I've been able to find online, this should
>> allocate a minimum of 512mb to the heap. or, maybe being a
>> novice to this myself, I'm misunderstanding something???
>>
>> On Thursday, July 31, 2014 3:52:53 AM UTC-7, Nando Breiter wrote:
>>

>> /The same sites ran flawlessly on a 6 year old Win2K/ACFMX
>> standard install server/
>> /
>> /

>> I've read somewhere in a thread about ColdFusion memory
>> issues that earlier versions of operating systems and
>> server software consumed less memory. If your setup worked
>> comfortably with 4GB on Win2K and ACF 6(?), 4GB may not be
>> sufficient with more recent releases ... again novice
>> speculation.
>>
>>
>>
>> Aria Media Sagl
>> Via Rompada 40
>> 6987 Caslano
>> Switzerland
>>
>> +41 (0)91 600 9601
>> +41 (0)76 303 4477 cell
>> skype: ariamedia
>>
>>
>> On Thu, Jul 31, 2014 at 10:15 AM, Nando Breiter
>> <na...@aria-media.com> wrote:
>>

>> /I am running Railo 4.2.1 / Tomcat 7.0.47 on Windows

>> 2008r2 (4GB) with IIS7.5 connecting via Boncode
>> connector. The system supports just over 200 contexts
>> for a total of 79 active websites in a production

>> environment./
>> /
>> /

>> it. 127.0.0.1:8009 <http://127.0.0.1:8009>". This

>> <https://groups.google.com/d/msgid/railo/6f641cc9-c3c9-4a64-a7be-ad6e08d7481a%40googlegroups.com?utm_medium=email&utm_source=footer

>> <https://groups.google.com/d/msgid/railo/6f641cc9-c3c9-4a64-a7be-ad6e08d7481a%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>>
>> > For more options, visit
>> https://groups.google.com/d/optout

>> <https://groups.google.com/d/optout>.

>>
>> --
>> Did you find this reply useful? Help the Railo
>> community and add it to the Railo Server wiki at
>> https://github.com/getrailo/railo/wiki
>> <https://github.com/getrailo/railo/wiki>
>> ---
>> You received this message because you are
>> subscribed to the Google Groups "Railo" group.
>> To view this discussion on the web visit

>> https://groups.google.com/d/msgid/railo/f4f81c09-92da-48b6-92ac-4e3175d97857%40googlegroups.com
>> <https://groups.google.com/d/msgid/railo/f4f81c09-92da-48b6-92ac-4e3175d97857%40googlegroups.com?utm_medium=email&utm_source=footer>.

>>
>>
>> For more options, visit
>> https://groups.google.com/d/optout

>> <https://groups.google.com/d/optout>.

>>
>>
>>
>> --
>> Did you find this reply useful? Help the Railo community and
>> add it to the Railo Server wiki at
>> https://github.com/getrailo/railo/wiki
>> <https://github.com/getrailo/railo/wiki>
>> ---
>> You received this message because you are subscribed to the
>> Google Groups "Railo" group.
>> To view this discussion on the web visit

>> https://groups.google.com/d/msgid/railo/ba8f7c05-08bd-4f02-8fd1-0a37faf42ddd%40googlegroups.com
>> <https://groups.google.com/d/msgid/railo/ba8f7c05-08bd-4f02-8fd1-0a37faf42ddd%40googlegroups.com?utm_medium=email&utm_source=footer>.

>> For more options, visit https://groups.google.com/d/optout

>> <https://groups.google.com/d/optout>.

>>
>> --
>> Did you find this reply useful? Help the Railo community and add it to
>> the Railo Server wiki at https://github.com/getrailo/railo/wiki
>> ---
>> You received this message because you are subscribed to the Google
>> Groups "Railo" group.
>> To view this discussion on the web visit

>> https://groups.google.com/d/msgid/railo/e50070b5-941a-4983-81de-112581e32ee3%40googlegroups.com
>> <https://groups.google.com/d/msgid/railo/e50070b5-941a-4983-81de-112581e32ee3%40googlegroups.com?utm_medium=email&utm_source=footer>.

>> For more options, visit https://groups.google.com/d/optout.
>
> --

> Igal Sapir
> Railo Core Developer
> http://getRailo.org/
>

> --
> Did you find this reply useful? Help the Railo community and add it to
> the Railo Server wiki at https://github.com/getrailo/railo/wiki
> ---
> You received this message because you are subscribed to the Google
> Groups "Railo" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/railo/53DA87D0.9080004%40getrailo.org
> <https://groups.google.com/d/msgid/railo/53DA87D0.9080004%40getrailo.org?utm_medium=email&utm_source=footer>.

[John Waddleton]

Well, I'm not sure why the Railo Server Properties utility was so hard for me to find. But now that I know that it exists and where to look for it, it all makes much more sense. And, adjusting the memory allocation seems to have solved my problem.

Since changing the settings yesterday, all the system indicators look really good. I have a high degree of confidence that this solved the problem!

Thank you everyone for your invaluable assistance!

[John Waddleton]

One final note for anyone that is following this thread...

Setting a timeout by including connectionTimeout="60000" in the server.xml file, as per Bilal's recommendation above, resulted in websites displaying a "Connection to Tomcat has been closed" error messages if left idle for - in retrospect not surprisingly -  only one minute. To the contrary, the BonCode user manual actually recommends removing the connectionTimeout setting entirely. So I removed it again... 

A quick refresh always brought the affected websites back, but it's a nuisance I and my customers would prefer to avoid for our website visitors. Now that I've removed the setting from server.xml, I'm crossing my fingers that my higher memory settings will still do the trick!

>>                         <a href="https://github.com/getrailo/railo/wiki" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Fgetrailo%2Frailo%2Fwiki\46sa\75D\46sntz\0751\46usg\75AFQjCNG89TXG0XPH9WuoB2J2K0KtglN-Lg';return true;" onclick="t

...