Have to restart Lucee every morning

2,271 views
Skip to first unread message

Jeramie Gatchell

unread,
Mar 16, 2015, 11:24:14 AM3/16/15
to lu...@googlegroups.com
Hello everyone, i'm extremely new to Lucee, just testing it out to see if we can use it instead of ACF as its so expensive.

I've put together a test server running:

Windows 2012 64bit (current on patches)
8gb ram
IIS 8
Tomcat 7.0.59 (installed during lucee installation)
Lucee 4.5.1 (upgraded to .008 from the dev path now)
MySQL 5.6.22


Every day sometime during the night the lucee server seems to crash and only responds with the following tomcat error:

Error connecting to Apache Tomcat instance.
Please check that a Tomcat server is running at given location and port.
Details:
No connection could be made because the target machine actively refused it 127.0.0.1:8009

You can change this message by changing TomcatConnectErrorURL setting in setting file.



If i stop and then restart the lucee server using the lucee server icon by the clock (actually i think this restarts tomcat?) then everything comes back up fine. for a while.

unfortunately i have absolutely no experience with apache, lucee, railo and java.  i've looked at all the logs i can find and there isn't anything in there at the time of the crash.  I have external monitoring every minute so i know the time that the server drops out.

the lucee server properties shows under the java tab
initial memory pool 256mb
max memory pool 512mb
thread stack size has no value entered

i am getting some errors at different times not near the time of the actual crash tho that possibly are contributing, i just don't know enough yet to have a clue. this first error looks promising tho, that error happens immediately after i restart the lucee server and memory usage does seem very high to me after a while.

  1. org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [] appears to have started a thread named [Thread-3] but has failed to stop it. This is very likely to create a memory leak.
  2. static resource /lucee/res/css/admin-1ea2d521fff0203b.css.cfm was not found @ C:\inetpub\wwwroot\WEB-INF\lucee\context\res\css\admin-1ea2d521fff0203b.css
  3. org.apache.tomcat.util.digester.SetPropertiesRule begin WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlNamespaceAware' to 'false' did not find a matching property.
  4. org.apache.tomcat.util.digester.SetPropertiesRule begin WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlValidation' to 'false' did not find a matching property.
Thank so much for any guidance you can give me.

Jeramie

Andrew Penhorwood

unread,
Mar 16, 2015, 1:02:31 PM3/16/15
to lu...@googlegroups.com
Strange situation.  I think we need more information to help.  I run Lucee on Linux and before that Railo.  My box might go months without being rebooted never had Lucee crash.  Have you watched the memory use with VisualVM or some other tool?

Andrew Penhorwood

Nando Breiter

unread,
Mar 16, 2015, 1:42:43 PM3/16/15
to lu...@googlegroups.com
It also smells of a memory issue to me, since you say you've allocated 8gb to the server, and only 512mb max to Lucee. I'd suggest trying to boost the initial and max memory values, doubling them at least, and see what happens.



Aria Media Sagl
Via Rompada 40
6987 Caslano
Switzerland

+41 (0)91 600 9601
+41 (0)76 303 4477 cell
skype: ariamedia

--
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/e2a78c62-21fc-4944-98c9-a951fe3cc5fa%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Jeramie Gatchell

unread,
Mar 16, 2015, 1:43:04 PM3/16/15
to lu...@googlegroups.com
Hi Andrew,

Thanks for the fast reply!

I installed VisualVM (i don't know anything about java unfortunately) but i'm not sure what i'm looking for.   i'm watching the heap monitor currently as it seems to have the most "action" (lol). i'll attach a screenshot.
monitor1037am.PNG

Jeramie Gatchell

unread,
Mar 16, 2015, 1:50:00 PM3/16/15
to lu...@googlegroups.com
Thanks Nando for the suggestion, i've gone into the "configure" option of the lucee server properties and changed initial memory pool to 512MB and maximum memory pool to 1024mb.  should there be a value in thread stack size? i have just accepted whatever the defaults are at this point.

rot...@smartway.cz

unread,
Mar 16, 2015, 2:15:05 PM3/16/15
to lu...@googlegroups.com
There are also some very cool plugins for VisualVM available through menu Tools > Plugins. I personally like "Tracer" ( visualizes what's going on over a longer period of time ) and "Memory Pools" ( shows detailed graphic view of different memory generations )  I have found these two quite helpful for troubleshooting issues like this.

Regards

Ivan Rotrekl

Igal @ Lucee.org

unread,
Mar 16, 2015, 2:16:06 PM3/16/15
to lu...@googlegroups.com
+1

Igal Sapir
Lucee Core Developer
Lucee.org

Igal @ Lucee.org

unread,
Mar 16, 2015, 2:18:30 PM3/16/15
to lu...@googlegroups.com
+1 for VisualVM Plugins

you should really start with the log files though.  specifically the Tomcat log files.  I don't know your setup, but by default they are usually at {Tomcat}/logs


Igal Sapir
Lucee Core Developer
Lucee.org

--
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.

Jeramie Gatchell

unread,
Mar 16, 2015, 2:32:59 PM3/16/15
to lu...@googlegroups.com
Thanks Ivan, i will check those two out.

rot...@smartway.cz

unread,
Mar 16, 2015, 3:38:20 PM3/16/15
to lu...@googlegroups.com
 Well, this might not be directly relevant to this case, but there are definitely some other settings which one might want to be aware of. I had to crank up maxThreads on AJP connector (lucee/tomcat/conf/server.xml), match MaxConnections in BondcodeAJP13.settings to it (lucee/AJP13/BondcodeAJP13.settings), increase -XX:maxPermSize in Java Options and switch to -XX:+UserG1GC before my server got finally stable. Simply, the default settings are too low for some and beg to be tweaked. (maxPostSize size in server.xml is another example of this).

Igal @ Lucee.org

unread,
Mar 16, 2015, 3:41:03 PM3/16/15
to lu...@googlegroups.com
what version of Java are you running?


Igal Sapir
Lucee Core Developer
Lucee.org

On 3/16/2015 12:38 PM, rot...@smartway.cz wrote:
 Well, this might not be directly relevant to this case, but there are definitely some other settings which one might want to be aware of. I had to crank up maxThreads on AJP connector (lucee/tomcat/conf/server.xml), match MaxConnections in BondcodeAJP13.settings to it (lucee/AJP13/BondcodeAJP13.settings), increase -XX:maxPermSize in Java Options and switch to -XX:+UserG1GC before my server got finally stable. Simply, the default settings are too low for some and beg to be tweaked. (maxPostSize size in server.xml is another example of this).
--
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.

rot...@smartway.cz

unread,
Mar 16, 2015, 3:56:39 PM3/16/15
to lu...@googlegroups.com
not sure if this was aimed at Jeramie or me: Just in case, mine is 1.7.0_76. I had started on lower version of JVM and Tomcat, which came with Railo about a year ago, though. Just recently I had installed Lucee, took advantage of the updated JRE and Tomcat and carried over my old settings...

Igal @ Lucee.org

unread,
Mar 16, 2015, 4:01:06 PM3/16/15
to lu...@googlegroups.com
aimed at Jeramie as I understand that he's the one experiencing the issues?

he seemed to have posted all of the versions except of the JVM, which is probably the most important one in an issue like that.


Igal Sapir
Lucee Core Developer
Lucee.org

On 3/16/2015 12:56 PM, rot...@smartway.cz wrote:
not sure if this was aimed at Jeramie or me: Just in case, mine is 1.7.0_76. I had started on lower version of JVM and Tomcat, which came with Railo about a year ago, though. Just recently I had installed Lucee, took advantage of the updated JRE and Tomcat and carried over my old settings...
--
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.

Chris Blackwell

unread,
Mar 16, 2015, 4:12:04 PM3/16/15
to lu...@googlegroups.com

I wondered this too, but his visual vm screenshot has a "meta space" tab, so it'll be 1.8

Igal @ Lucee.org

unread,
Mar 16, 2015, 4:17:27 PM3/16/15
to lu...@googlegroups.com
good catch.  so PermSize is not an issue in his case.

most likely has to increase the heap size, but a glance at the log files will be revealing.


Igal Sapir
Lucee Core Developer
Lucee.org

Jeramie Gatchell

unread,
Mar 21, 2015, 11:33:21 PM3/21/15
to lu...@googlegroups.com
i'm so sorry for not replying, i thought the system would email me when there was a reply to the thread! woops!

i am indeed running jvm 1.8.

i'm going to attach the log files for the 19th.  The server crashed at 7:34pm and started responding with the error in the op.  at 10:23pm i stopped and started the lucee server.
commons-daemon.2015-03-19.log
localhost_access_log.2015-03-19.txt
catalina.2015-03-19.log
lucee-stdout.2015-03-19.log
lucee-stderr.2015-03-19.log

Jeramie Gatchell

unread,
Mar 21, 2015, 11:43:25 PM3/21/15
to lu...@googlegroups.com
here also are the server.xml file and the BonCodeAJP13.settings files, i've sanitized them i hope.. i do have 3 websites running on this server, but they are freebie ones i host for friends that get 1 or 2 hits a month if they are lucky... 
BonCodeAJP13.settings
server.xml

Jeramie Gatchell

unread,
Mar 22, 2015, 12:05:50 AM3/22/15
to lu...@googlegroups.com
Hi Ivan,

Thanks so much for replying.  I'm absolutely 100% green when it comes to java/tomcat/etc.. so i've bitten off a bit more than i can chew right now.  i'm trying to follow what you are saying below but unfortunately i'm struggling.

i'm looking in lucee/tomcat/conf/server.xml file and found the below entry:
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" /> 

i looked in the bondcodeajp13.settings file and maxconnections is set to 200 so i've added maxThreads="200" to the ajp connector line, which now looks like this:
<Connector port="8009" protocol="AJP/1.3" maxThreads="200" redirectPort="8443" />

Please correct me if that is incorrect.

Now, the java options you reference have completely gone over my head.  1st, i'm not even sure where to change that, i see a java tab when i click on the configure option of the lucee service manager icon (apache server manager?, the little icon down by my clock) on the java tab in the java options textbox is the following:

-Dcatalina.base=C:\lucee\tomcat
-Dcatalina.home=C:\lucee\tomcat
-Djava.endorsed.dirs=C:\lucee\tomcat\endorsed
-Djava.io.tmpdir=C:\lucee\tomcat\temp
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Djava.util.logging.config.file=C:\lucee\tomcat\conf\logging.properties
-javaagent:C:\lucee\tomcat\lib\lucee-inst.jar

is this the correct place? and if so, what would be a decent value to start with and (lol i know, i'm a newb) what would be the exact line?

Thanks so much, i truly appreciate the assist with this learning curve.

Ivan Rotrekl

unread,
Mar 22, 2015, 10:34:15 AM3/22/15
to lu...@googlegroups.com
Hi Jeramie,

I'm  definetely no expert at this subject, but did you try to increase the "Initial memory pool" and "Maximum memory pool" values in the Java tab in your Lucee service manager ?
You also should be able to see in your VisualVM if the memory reached some limit.

Also your  lucee-stderr.log finishes like this:

SEVERE: The web application [] appears to have started a thread named [Thread-31] but has failed to stop it. This is very likely to create a memory leak.
Mar 20, 2015 10:18:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [] appears to have started a thread named [Thread-3] but has failed to stop it. This is very likely to create a memory leak.
Mar 20, 2015 10:18:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] appears to have started a thread named [Thread-16] but has failed to stop it. This is very likely to create a memory leak.

I'm not sure what that means. But I  guess if there is memory problem, increasing the memory available to tomcat could at least postpone it and allow more time for investigation.

Maybe, you could also try to create some screenshots of the VisualVM at the time of crash,  if the server crashes again :)

Regards

Ivan




--
You received this message because you are subscribed to a topic in the Google Groups "Lucee" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lucee/1MzfFUOAeeo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lucee+un...@googlegroups.com.

To post to this group, send email to lu...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
S pozdravem

Ivan Rotrekl

SMARTWAY s.r.o. |
|Grafický design | Web design | Reklamní fotografie | Produkce |

|Sportovní 457| Vestec - Praha západ | 25242 |

Ivan Rotrekl

unread,
Mar 22, 2015, 11:30:49 AM3/22/15
to lu...@googlegroups.com
Also, you have 8 GB RAM available.

This might be a very simplistic and unscientific advice, but you can simply open your "Reliability and Performance Monitor", see how much memory the system consumes and give the rest to Tomcat, until you reach somewhere between 70 - 80 % of memory consumption ( in the long run ).

In my case I have IIS 7.5, windows 2008 R2 and 10 GB RAM available. I gave to Tomcat 4 GB and the server is running around 80 - 85 % RAM overall. In the VisualVM I can see, the heap size peaks from 2GB to 3.7GB. I was thinking free memory is a wasted memory.

This worked for me so far. But I am aware that it is very crude way to deal with a much more complex subject.

Regards

Ivan

Jeramie Gatchell

unread,
Mar 23, 2015, 12:31:19 AM3/23/15
to lu...@googlegroups.com
I set the max 4gb to tomcat for now and we'll see how it does.  so far it hasn't crashed in over 24 hours =]

inside visualvm the max heap size appears to be 256MB and the current heap hasn't exceeded about 150mb (so far) with used fluctuating between 25mb and 75mb.. but i'll keep an eye on it and see.

thank you very much everyone for the awesome community help!  let me know if there is anything else you guys can think of that i might want to check out. 

Pritesh Patel

unread,
Mar 23, 2015, 9:33:11 AM3/23/15
to lu...@googlegroups.com
Hi Jeramie,
After looking screenshot I realize you are not monitoring Lucee instead it was stats for Visulavm itself. You may need add one JMX entry for Lucee as well. I had wrote blog in past for setup with ColdFusion but it will be similar with Lucee as well.

Just give try and may be you want to add some more plugin as suggested by others.

Thanks,
Pritesh

Jeramie Gatchell

unread,
Mar 23, 2015, 3:47:53 PM3/23/15
to lu...@googlegroups.com
oh wow, you are correct Pritesh, thank you.  I was indeed just monitoring visualvm itself..

Using your blog i was able to add lucee and see far different numbers (lol, imagine that!).

So far, the heap size is sitting around 1.75GB and the heap used is at about 1.1GB and slowly rising.  I started a tracer on it and will see what happens if/when it crashes next.  its been up nearly 48 hours so far, thats the longest its stayed alive for me so far.


Jason Tully

unread,
Mar 24, 2015, 12:05:49 PM3/24/15
to lu...@googlegroups.com
We used to have the same issue with Railo last year. I would have to restart every day. Since tweaking, Railo never crashes or needs to be restarted. You can see my post here (https://groups.google.com/forum/#!topic/railo/FAd4uXOddJY)  with suggestions. Some of our settings have changed since then, attached is screenshot of what my settings are at now, and they are rock solid. Another key is to have your maxThreads in your Tomcat server.xml at a high number. We used to have it at 500, but after a week, the slots would fill up thus locking up the server making Railo seem unresponsive (when it really wasn't)  We have since changed it to 1500, and haven't had an issue since. YOu can see how many threads you are using in your tomcat web interface.

<Connector port="8009" acceptorThreadCount="6" maxThreads="1500" minSpareThreads="150" maxSpareThreads="500"  tcpNoDelay="true" enableLookups="false" protocol="AJP/1.3" redirectPort="8443" />

My setup is 

Windows Server 2012 (R1) with 32gb Ram
IIS8 using BonCode
Mod_CFML with TomCat 7

We haven't upgrade our production box to Lucce yet, but will be doing so this month.
railo-props.png

Ivan Rotrekl

unread,
Mar 24, 2015, 1:03:43 PM3/24/15
to lu...@googlegroups.com
Hi Jason,

after seeing your snippet, I was wondering: Is there some reason to increase the number of minSpareThreads ?  I'm not challenging it,  just trying to find out.

I've been running it like this:

<Connector port="8009" protocol="AJP/1.3" maxThreads="1000" enableLookups="false" minSpareThreads="4" redirectPort="8443" maxPostSize="52428800"/>

In VisualVM can see   most of the time around 70-300 threads being created.

According to  http://www.genericarticles.com/mediawiki/index.php?title=How_to_optimize_tomcat_performance_in_production, it should work like this:
At server startup, the HTTP Connector will create a number of processing threads based on the value configured for the minSpareThreads attribute.
   * Each incoming request requires a thread for the duration of that request.
   * If the number of simultaneous requests cannot be handled by the currently available request processing threads, additional threads will be created up to the configured maximum (the value of the maxThreads attribute).
   * If still more simultaneous requests are received, they are stacked up up to the configured maximum (the value of the acceptCount attribute).
   * Any further simultaneous requests will receive "connection refused" errors, until resources are available to process them.
So, is it better to create that 150 minSpareThreads up front at the server startup, rather than wait for Tomcat to create them one by one as needed ?

Regards

Ivan

--
You received this message because you are subscribed to a topic in the Google Groups "Lucee" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lucee/1MzfFUOAeeo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Terry Whitney

unread,
Mar 24, 2015, 4:26:11 PM3/24/15
to lu...@googlegroups.com
This is a windows IIS issue, not a Tomcat / Lucee / Railo / ACF issue.


I have seen this numerous times where the same code on the same physical or virtual servers requires the Windows boxes to reboot where as the Linux / NIX boxes are fine.


The memory leak is dependent upon IIS Services, the version of Java you have running, and possibly the use or lack of use of error correcting memory. 

The easiest thing to do is just run a restart services script or reboot script during off hours. If you can not do that, as its an always up, always online site,  your choices are to either use an alternative install, or fire up at least one other server and start load balancing.




On Monday, March 16, 2015 at 11:24:14 AM UTC-4, Jeramie Gatchell wrote:

Igal Sapir

unread,
Mar 24, 2015, 11:34:58 PM3/24/15
to lu...@googlegroups.com

I've been running ACF and Railo on Windows for years without having any problem.

This is Not a "Windows issue".

--
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.

To post to this group, send email to lu...@googlegroups.com.

Bilal

unread,
Mar 25, 2015, 11:58:17 AM3/25/15
to lu...@googlegroups.com
I would suggest the following elements:
a) Timeout Idle AJP Conennections using this attribute in server.xml AJP declaration (e.g. to 3 mins). 
connectionTimeout="180000"   

b) When connection errors occur use a nicer message or trigger a task (even restart) using the automatic redirect URL from connector. For example, add a connect error url setting to the BonCode setting file: TomcatConnectErrorURL to forward users to a nicer message etc.:
<TomcatConnectErrorURL>http://mysite/errorhandler.jsp</TomcatConnectErrorURL>
%3CTomcatConnectErrorURL%3Ehttp%3A%2F%2Fmysite%2Ferrorhandler.jsp%3C%2FTomcatConnectErrorURL%3E

HTH,
Bilal
 

Jeramie Gatchell

unread,
Mar 26, 2015, 1:10:53 PM3/26/15
to lu...@googlegroups.com
does anyone the minimum memory recommended for a lucee install on windows?

we are a tiny development company and don't have anything with 32gb.. we have ACF virtual machines with 2gb of ram that run great.

Jeramie Gatchell

unread,
Mar 26, 2015, 1:12:42 PM3/26/15
to lu...@googlegroups.com
Thanks Jason, i will try this.  the previous changes i made with regard to increasing the max mem pool to 4gb only delayed the crashes.  it takes 2 or 3 days to crash now. i've moved it back down to 2gb, one because it should be able to run with that much to be viable for us and second to speed up the crashing so i know if something worked or not.

Igal @ Lucee.org

unread,
Mar 26, 2015, 1:18:34 PM3/26/15
to lu...@googlegroups.com
please post the Tomcat log files somewhere -- catalina-out and catalina-err.


Igal Sapir
Lucee Core Developer
Lucee.org

--
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.

Jeramie Gatchell

unread,
Mar 26, 2015, 1:43:02 PM3/26/15
to lu...@googlegroups.com
in lucee/tomcat/logs i see lucee-stderr and lucee-stdout logfiles, would those be the right ones?

Martin Schaible

unread,
Mar 26, 2015, 1:46:59 PM3/26/15
to lu...@googlegroups.com

we have ACF virtual machines with 2gb of ram that run great
Did i read that correctly? You're running a windows box with a total of 2GB RAM and serving a Lucee environment?

I have posted an example of a memory configuration for Tomcat here:

https://bitbucket.org/lucee/lucee/wiki/Configuring%20Tomcat%20as%20a%20Windows%20service

The configuration runs on some servers with no problems.

If your box has less memory, decrease the settings accordingly.

Igal @ Lucee.org

unread,
Mar 26, 2015, 1:49:32 PM3/26/15