Lucee stops serving pages and needs to be restarted every day.

120 views
Skip to first unread message

arkleyjoe

unread,
Sep 27, 2016, 10:05:18 AM9/27/16
to Lucee
Hi,

We have a situation where Lucee stops serving pages and we constantly have to restart the service to get going again. Most of the time a simple restart will do the trick. I am reluctant to say Lucee "crashes" because the service still seems to be running and I wonder if the problem with with Tomcat, mod_CFML, ajp or some other part of the set up. 

From time to time if we are under load it will fall over multiple times again within a few minutes but usually, it crashes approximately every 24 hours, we restart and all is fine again.

We have two identical servers running and both of them crash in exactly the same way. It is not uncommon for both to crash taking us offline entirely.

Usually there is no particular error displayed and the Lucee service seems to still be running. Sometimes it can take a significant amount of time to stop the Lucee service but start up is usually pretty quick.

We have FusionReactor installed on one of the servers and that has not really allowed us to get any meaningful insights and we have not found anything of particular note in the logs. 

Around the time of the crashes, we do see timeouts but it is hard to tell if these are the cause of the problem or if they are a symptom of the problem.

Any guidance would be much appreciated as I do not really know why to try next.

Here are some environment / config settings:

OS: Windows Server 2012 (6.2) 64bit 
Servlet Container: Apache Tomcat/8.0.24 
Java: 1.8.0_45 (Oracle Corporation) 64bit  
Architecture: 64bit  

JVM:
Initial Memory Pool : 1024mb
Max Memory Pool: 4096mb





Harry Klein

unread,
Sep 27, 2016, 10:20:16 AM9/27/16
to Lucee

Which Lucee Version are you using?

Early Lucee 5 Versions had some severe problems.

 

-Harry

--
Get 10% off of the regular price for this years CFCamp in Munich, Germany (Oct. 20th & 21st) with the Lucee discount code Lucee@cfcamp. 189€ instead of 210€. Visit https://ti.to/cfcamp/cfcamp-2016/discount/Lucee@cfcamp
---
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/d321d191-184a-4a2f-b79a-6c4ade9c1146%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

arkleyjoe

unread,
Sep 27, 2016, 10:23:16 AM9/27/16
to Lucee
Hi Harry

Version Lucee 4.5.2.018 final (sorry should have posted that with the original)

Joe

Igal @ Lucee.org

unread,
Sep 27, 2016, 12:15:25 PM9/27/16
to lu...@googlegroups.com

Joe,

The first thing I'd do is update to the latest on the 4.5 branch (currently 4.5.3.018) from http://lucee.org/downloads.html

The next thing to check is, can you hit Tomcat directly when the issue happens?  i.e. on port 8080 or 8888 depending on your setup.  That will take IIS and the connector out of the equation, so please do that and let us know your findings.


Igal Sapir
Lucee Core Developer
Lucee.org

--
Get 10% off of the regular price for this years CFCamp in Munich, Germany (Oct. 20th & 21st) with the Lucee discount code Lucee@cfcamp. 189€ instead of 210€. Visit https://ti.to/cfcamp/cfcamp-2016/discount/Lucee@cfcamp
---
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.

arkleyjoe

unread,
Sep 27, 2016, 12:22:04 PM9/27/16
to Lucee
Hi Igal,

Thanks for coming back to me so quickly. 

Let me look into the upgrade this afternoon. Just so I can be sure what to expect when hitting Tomcat directly. I will need to test for sure but I believe we are able to reach the Lucee Admin pages on 8888 when the server has stopped serving pages on port 80. Would the admin panel still being served indicate that Tomcat is still running?

Joe

Igal @ Lucee.org

unread,
Sep 27, 2016, 12:44:03 PM9/27/16
to lu...@googlegroups.com

Well, the reason we test "Tomcat direct" vs "Tomcat via IIS" is to eliminate IIS and the connector as the cause of the problem, so if a page is not accessible via Tomcat-direct, we know that the issue is not with IIS+Connector. 

But on the other hand, if a page is not accessible via IIS+Connector, but is accessible via Tomcat-direct, then we know that the issue is not with Tomcat/Lucee, but is with IIS/Connector.

Think of this as separating the system into two parts and checking if the problem is in the first part (IIS+Conector) or the second part (Tomcat+Lucee).

Being able to access the Admin on port 8888 when the service hangs could be an indication that the issue is indeed with IIS/Connector, but I am not familiar with your setup so it's best to test on the same page when the service hangs.

Igal Sapir
Lucee Core Developer
Lucee.org

arkleyjoe

unread,
Sep 27, 2016, 1:05:53 PM9/27/16
to Lucee
Hi Igal,

No problem your reason for that approach was clear. I was just not 100% sure if the admin tool access would prove the case as I was already aware how that behave. 

I will need to wait for it to happen again and test the service on the other port and revert.

Joe 

Igal @ Lucee.org

unread,
Sep 28, 2016, 2:35:21 AM9/28/16
to lu...@googlegroups.com

I was just not 100% sure if the admin tool access would prove the case as I was already aware how that behave.

in general yes, the Admin is just another CFML application so it should prove that the issue is not with Tomcat/Lucee if you are able to access it, but to prove that point -- try to access the Admin via IIS, so if on IIS you are blocking the Admin for security purposes, for example, then you can not really test it that way, but if in general when the server is responsive you can access the Admin both via IIS and via Tomcat-directly, and then when it's unresponsive you can access it from Tomcat-directly but not from IIS, then the issue is not with Tomcat/Lucee, but rather with IIS/Connector.  (sorry for the convoluted answer.  long day.)

Igal Sapir
Lucee Core Developer
Lucee.org

--
Get 10% off of the regular price for this years CFCamp in Munich, Germany (Oct. 20th & 21st) with the Lucee discount code Lucee@cfcamp. 189€ instead of 210€. Visit https://ti.to/cfcamp/cfcamp-2016/discount/Lucee@cfcamp
---
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.

arkleyjoe

unread,
Sep 28, 2016, 11:20:51 AM9/28/16
to Lucee
Hi Igal,

Sorry for taking a while to reply. I had to wait for it to crash again naturally in order to be able to test.

I can confirm that when this happens I am able to hit the pages on port 8888 while they fail with no apparent error on port 80 via IIS. So I guess this confirms that our problem is related to the connector.

Around the time this happened we did also get a small number of blocked threads, although I was not able to get a trace.

I would welcome your thoughts on how we may address this?

Kind Regards

Joe  

Igal @ Lucee.org

unread,
Sep 28, 2016, 1:03:17 PM9/28/16
to lu...@googlegroups.com

OK, cool.  So now we confirmed that the problem is with IIS/mod_cfml.

IIS is used widely by many users, so if I have to guess which one of those components has a concurrency issue I would put my money on the mod_cfml.  mod_cfml for Windows is using the BonCode connector -- http://www.boncode.net/ -- so first make sure that you're running the latest version of that, and try to troubleshoot by logging requests etc.

You can also start a new thread (or simply change the subject line) with words like BonCode or mod_cfml with IIS, and Bilal or Jordan will probably notice it and chime in.

Another alternative is to switch from IIS.  I know that it may "sound scary" to give up the GUI administration tool, but I've made that leap myself several years ago and switched to nginx, and after a short period of time of getting familiarized with the new web server I've never had issues with the web server again.

Best,

Igal Sapir
Lucee Core Developer
Lucee.org

--
Get 10% off of the regular price for this years CFCamp in Munich, Germany (Oct. 20th & 21st) with the Lucee discount code Lucee@cfcamp. 189€ instead of 210€. Visit https://ti.to/cfcamp/cfcamp-2016/discount/Lucee@cfcamp
---
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.

arkleyjoe

unread,
Sep 28, 2016, 1:28:47 PM9/28/16
to Lucee
Igal, thank you very much for your help.

arkleyjoe

unread,
Sep 28, 2016, 1:47:00 PM9/28/16
to Lucee
Igal, thank you very much for your help.

For anyone wishing to follow the other thread.

Reply all
Reply to author
Forward
0 new messages