Lucee Performance/caching

[Scott Conklin]

On a recent upgrade from ACF to Lucee we have noticed a situation where performance becomes an issue that we did not have on ACF. 

We notice that under normal operations, our sites run very fast, faster than they did on an equivalent machine running ACF 9.01. 

Our application architecture consists of two parts, and application builder and the applications built by the builder. 

The application builder reads our standard templates (about 160 or so) into memory and replaces various mnemonics with client-customized snippets of codes and writes the .cfm pages to the webroot for each of our 55  sites running on our Lucee 4.5 installation. (win 2012)

The builder can loop over all 55 clients and rewrite all the pages of those sites in about 4 or 5 minutes. 

Over the years, we have always done this throughout regular business hours without any issue...sometimes several times a day.

In Lucee, we have found that while the application builder is running,  access to the sites comes to almost a screeching halt ..taking sometimes 20, 30 or 40 seconds for pages to complete.  and on some occasions the pages timeout .. with (sometimes) the following errors appearing the log files.

java.lang.ThreadDeath

at java.lang.Thread.stop(Unknown Source)

at lucee.commons.io.StopThread.run(SystemUtil.java:1091)

Exception in thread "Thread-53889" java.lang.NullPointerException

at lucee.commons.io.StopThread.run(SystemUtil.java:1068)

Exception in thread "ajp-nio-8009-exec-14" java.lang.IllegalMonitorStateException

at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(Unknown Source)

at java.util.concurrent.locks.ReentrantLock.unlock(Unknown Source)

at java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)

at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)

at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

at java.lang.Thread.run(Unknown Source)

It does recover eventually.. maybe 3 or 4  minutes after the application builder completes, but not before someone complains of performance issues by our clients.

I have tried all 3 settings for Inspect Templates (CFM/CFC)  setting (always, once, never) 

The problem occurs exactly the same for always and once.

Obviously, when set to never it the issue goes away but as soon as  I click "Clear page cache Pool"  to make the latest rebuild changes persistent, the issue shows itself . This leads me to speculate that it must have something to do with the algorithms used to populate/refresh the cache or  the comparison routine  for large-scale wholesale changes in the file timestamps? 

Hate to bring up the fact that it the behavior was different in ACF  but this is something we need to  be able to do a regular basis without interference to our clients 

ability  to work. 

Has anyone else ever had such an experience? 

Any thoughts on why this happens? and what we can do about it? 

[Scott Conklin]

anyone? 

[Andrew Dixon]

Hi Scott,

Can you give some more details of your set up, e.g.:

* JVM Settings

* JVM Version

* Servlet container being used

* Web server being used

Thanks.

Kind regards,

Andrew

about.me - mso - Lucee Association Member

--
Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html
---
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/37ba3e71-562e-4c81-83a3-3474b789c643%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Brad Wood]

If I understand correctly, are the timestamps being updated for every single file?  We had the same issue when our Jenkins deploy did this as every template needed to be recompiled on the next request.  We had to modify our process to not update timestamps on file which did not change.

Maybe Micha can comment on the compilation performance in Lucee.

Thanks!

~Brad

[Scott Conklin]

@andrew 

sure.. 

OS

Windows Server 2012 R2 (6.3) 64bit 

8 GB RAM

Servlet Container

Apache Tomcat/8.0.28 

Java

1.8.0_77 (Oracle Corporation) 64bit  

Architecture

64bit  

Database

MS SQL Server 2014 

Db Driver

jTDS Type 4 JDBC Driver for MS SQL Server 

Java args:

-Xms4096m -Xmx4096m 

On Monday, May 9, 2016 at 5:30:38 PM UTC-5, Scott Conklin wrote:

[Andrew Dixon]

@brad - would that be this ticket:

https://luceeserver.atlassian.net/browse/LDEV-638

Kind regards,

Andrew

about.me - mso - Lucee Association Member

--
Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html
---
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/d90975ec-0be0-488c-a06d-bec5ecc52a0d%40googlegroups.com.

[Igal @ Lucee.org]

what if you add a call like `Sleep(10)` in the tight loop of the Builder application?  it will elongate the build time but possibly make the server more responsive?

Igal Sapir
Lucee Core Developer
Lucee.org

[Scott Conklin]

@Brad-

I believe that would correct.  because the application builder that i talk about  reads the standard templates and then re-writes to disk all the .cfm files into the webroot.

There are about 58 webroots (one for each of our clients) and the number of files  in each webroot is about 166 files. 

see an example of a webroot datetime stamp of the files after the application builder as rewritten them all.. 

http://www.evernote.com/shard/s349/sh/28c91a6e-1eea-4284-9581-862a8e832c6c/9f1ba6d2fdfa4abebc28021500d18001

On Monday, May 9, 2016 at 5:30:38 PM UTC-5, Scott Conklin wrote:

[Mark Drew]

This has always been a bit of an issue with Lucee

The way to solve it is slightly more complicated but it might work. 

A) All your applications should be a mapping , so for example your generated templates should be accessed from the mapping /generated that points to some folder. Let’s call it “generated_a"

B) When you are building your templates, build them into a folder called “ generated_b”. When the process stops, tell that site to update the /generated mapping from generated_a to generated_b

C) next time you build swap them mapping from generating_b to generating_a

Long and short of it, dumping a bunch of templates kicks off a process of compilation, which ultimately makes Lucee faster. Except in this case. I am sure there are better explanations (about Heap and non-heap memory and class loaders) but that is a fact. 

This way you can be a bit gentler on Lucee :) 

If you are doing LOADS of sites (you say 55?) are they also each a context (with it’s own WEB-INF) or are you doing something else?

HTH

MD

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/ec10677a-7449-02e4-9c50-a7479a57696c%40lucee.org.

[Scott Conklin]

>> if you disable the java agent, then the slow down is not massive, it becomes normal.

What does this mean  

On Monday, May 9, 2016 at 5:30:38 PM UTC-5, Scott Conklin wrote:

[Mark Drew]

in the setenv.sh file under /tomcat/bin there will be a -javaagent: entry that maps to a lib. You could copy the line, comment out the duplicate and remove the javaagent entry

MD 

-- 
Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html
--- 
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/3caeae94-4650-4843-9fb9-96d20c22df96%40googlegroups.com.

[Brad Wood]

I've been omitting the java agent on some of my 4.5 servers now that I'm on java 8.  The biggest thing it does is it allows the class files to be rewritten in place on recompilation so perm gen doesn't fill up.  On Java 8, perm gen doesn't exist and classes can be GC'd.  I've also run into some really weird compilation bugs in Lucee which were caused by the java agent. I'm curious to know what difference it makes for you to remove your java agent.

Thanks!

~Brad

ColdBox Platform Evangelist

Ortus Solutions, Corp 

E-mail: br...@coldbox.org

ColdBox Platform: http://www.coldbox.org 

Blog: http://www.codersrevolution.com

You received this message because you are subscribed to a topic in the Google Groups "Lucee" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lucee/56WHZlWc9rI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lucee+un...@googlegroups.com.

To post to this group, send email to lu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/30A92C53-F98D-4D22-8799-DA8C1B20355B%40gmail.com.

[Scott Conklin]

@mark-

>> If you are doing LOADS of sites (you say 55?) are they also each a context (with it’s own WEB-INF)

Yes, each is its own context. (all have their own WEB-INF) 

I think you are saying to alter the builder to write the files to one of two possible webroot paths and when the building is done, just toggle the mapping programatically 

to the other mapped webroot? is that correct? 

I would need to have  something like webroot_clientX_A and webroot_ClientX_B for all 55 clients? 

[Gert Franz]

Hey Scott,

 

What is important to know:

·         Why is the process taking 4-5 minutes? Writing around 10000 files shouldn’t be that much of work

·         Where are the requests hanging? There is a way to find that out (dm or email me for details)

·         What is the exact configuration of the server doing the build?

 

I would have more questions once the above are off the table J

Normally Lucee should really perform a lot faster than that. IMHO it could be a configuration or locking issue. Nonetheless I will help you as good as I can.

 

Sincerely
Gert Franz

 

RASIA GmbH

Spittelgasse 7

5103 Moeriken-Wildegg

Switzerland

Email: ge...@rasia.ch
Skype: gert.franz

Phone Switzerland: +41 76 5680 231

 

--

Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html
---
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/37ba3e71-562e-4c81-83a3-3474b789c643%40googlegroups.com.

[Gert Franz]

Oh the thread death btw. could be Lucee killing long running requests.

 

Sincerely
Gert Franz

 

RASIA GmbH

Spittelgasse 7

5103 Moeriken-Wildegg

Switzerland

Email: ge...@rasia.ch
Skype: gert.franz

Phone Switzerland: +41 76 5680 231

 

 

Von: lu...@googlegroups.com [mailto:lu...@googlegroups.com] Im Auftrag von Scott Conklin
Gesendet: Mittwoch, 11. Mai 2016 18:12
An: Lucee
Betreff: [Lucee] Re: Lucee Performance/caching

 

anyone? 

--

[Mark Drew]

Sure, (I replied before you emailed your structure) 

the problem here would be that you are going to have to tell each web context to change it’s mapping. 

You should try the javaagent path and see if that improves the process. Otherwise you would have to write the following process (some of which might be a little complicated) 

1) Build process starts, go get that application/site’s webINF and find the mapping. Get which folder is the root. 

2) clear out the ALTERNATE root (so if current root is A, clear out B) 

3) write to the alternate root, 

4) once done, tell the application to switch it’s mapping from A to B

MD

--
Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html
---
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/e6c838e9-0473-4cb2-9a20-bd58586e3cd1%40googlegroups.com.

[Scott Conklin]

@mark @brad
Thanks, I am going to give that a try.... 

On Monday, May 9, 2016 at 5:30:38 PM UTC-5, Scott Conklin wrote:

[Scott Conklin]

@mark-

Another quick question.. 

When I follow your instructions  for modifying the Setenv.sh 

I end up with this 

CATALINA_OPTS="-Xms256m -Xmx512m";

# CATALINA_OPTS="-Xms256m -Xmx512m -javaagent:lib/lucee-inst.jar";

1)  this is what you meant? correct? 

2)  curious what these xms and xmx  setting here are when i have  

these in my java args for Lucee 

-Xms4096m -Xmx4096m 

3) I am assuming that i will need to cycle Lucee to make this change persist?
(is so, I may have to wait until after business hours to try this out)

[Mark Drew]

Not sure which way round it goes (not a windows guy myself) but these are default Max and Min memory settings. The file looks correct BTW. 

MD 

--
Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html
---
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/6e8a8568-2136-4c76-ade2-93aed4ac9276%40googlegroups.com.

[Gert Franz]

Well hopefully with higher xmx and xms values J

 

Sincerely
Gert Franz

 

RASIA GmbH

Spittelgasse 7

5103 Moeriken-Wildegg

Switzerland

Email: ge...@rasia.ch
Skype: gert.franz

Phone Switzerland: +41 76 5680 231

 

 

Von: lu...@googlegroups.com [mailto:lu...@googlegroups.com] Im Auftrag von Scott Conklin
Gesendet: Mittwoch, 11. Mai 2016 19:23
An: Lucee
Betreff: [Lucee] Re: Lucee Performance/caching

 

@mark-

--

[Scott Conklin]

@gert 

>> Well hopefully with higher xmx and xms values

What do you actually mean?


My min max values are set for Lucee in the Tomcat Service Control Manager
See the screenshot
http://www.evernote.com/shard/s349/sh/e93a7ee2-f6eb-4184-8bb6-dd3e7d0f7ea2/5abb05e56ca54a3bdbf53a9a618391bd

correct? 

My previous question is what do these values mean in the setenv.sh?

CATALINA_OPTS="-Xms256m -Xmx512m;
# CATALINA_OPTS="-Xms256m -Xmx512m -javaagent:lib/lucee-inst.jar";

Should i be making these values consistent with the ones in the Tomcat Service Control Manager? 

I have never even heard the setenv.sh file until @mark pointed it out to me in this thread 

[Scott Conklin]

on further inspection, i now see that the java options in the Tomcat Service Control Manager also has the javaagent flag in it

Should i be removing it here or in setenv.sh? 

is the setenv.sh the way one would do if on a non-windows machine? or does one have a precedence over the other? 

http://www.evernote.com/shard/s349/sh/e938c0b5-5ad8-49ff-b311-6e75655ea6f9/cf2fc4670d7872602ad32bde46f8fa23

[Brad Wood]

If you're on Windows, I don't think setenv.sh is used.  On Linux, setenv.sh is used for any java process that's spawned-- including the process that stops the server.  There is a catalina.sh file that controls the specific args when starting up the web server.  On Windows, it all uses that lame little GUI box.  You should be fine to just remove it there.

-Xms is "memory start" size for the heap

-Xmx is "memory max" size for the heap

Conventional wisdom is to keep those values the same on a server so the JVM doesn't need to bother asking the OS for more RAM. On a dev machine, I make the start size smaller so it doesn't waste RAM it doesn't need.

Thanks!

~Brad

ColdBox Platform Evangelist

Ortus Solutions, Corp 

E-mail: br...@coldbox.org

ColdBox Platform: http://www.coldbox.org 

Blog: http://www.codersrevolution.com

--
Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html
---

You received this message because you are subscribed to a topic in the Google Groups "Lucee" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lucee/56WHZlWc9rI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lucee+un...@googlegroups.com.

To post to this group, send email to lu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/e06d57cb-5a40-49e1-8cf7-72dcf2a689ed%40googlegroups.com.

[Igal @ Lucee.org]

On Windows you set it through the Service Applet, not through setenv, and definitely NOT through setenv.sh which is a *nix shell script.

And stick to your 4GB of RAM rather than the default 256MB -- 512MB settings.

Igal Sapir
Lucee Core Developer
Lucee.org

--
Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html
---

You received this message because you are subscribed to the Google Groups "Lucee" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.

To post to this group, send email to lu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/6e8a8568-2136-4c76-ade2-93aed4ac9276%40googlegroups.com.

[Mark Drew]

Sorry, I gave you bad info at the time, setenv.sh is just for ‘nix type systems. Stick to your service control thing. 

MD 

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/6a7d4e4a-1135-6c12-8225-f33131c425ae%40lucee.org.

[Scott Conklin]

Thank you all! 

I will try removing the -javaagent:E:\lucee\tomcat\lib\lucee-inst.jar setting from the control panel and the run the application builder again to see if it stops the slow down symptoms

I will have to wait until after business hours to do so to avoid service interruptions to our corporate clients.. 

Will let you know how the test goes... 

Thanks again!

[Gert Franz]

Ok, and let us know how it worked out but I am more interested why generating 9000 files takes 5 minutes???

 

Sincerely
Gert Franz

 

RASIA GmbH

Spittelgasse 7

5103 Moeriken-Wildegg

Switzerland

Email: ge...@rasia.ch
Skype: gert.franz

Phone Switzerland: +41 76 5680 231

 

 

Von: lu...@googlegroups.com [mailto:lu...@googlegroups.com] Im Auftrag von Scott Conklin
Gesendet: Mittwoch, 11. Mai 2016 20:02
An: Lucee
Betreff: [Lucee] Re: Lucee Performance/caching

 

Thank you all! 

--

Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html
---
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/d19e9d7d-82d3-40d3-9c5d-aa77cdb1303f%40googlegroups.com.

[Scott Conklin]

sorry for talking so long to respond on my findings on this issue. I was out last week.

I am happy to report that implementing the suggestion made by @brad ; (removing this -javaagent:C:\lucee\tomcat\lib\lucee-inst.jar from the java arguments) 

completely solved this issue.. 

All through the application building process the site loads times remain quick and unaffected by rewriting all the files of all the websites to their respective webroots.

AND under Lucee, the application builder itself is now running faster than I have ever seen it averaging under a second for each venue (website) (took about 3 seconds under ACF) 

Thanks so much for everyone's input on this.

[David Eurenius]

What exactly does the -javaagent:C:\lucee\tomcat\lib\lucee-inst.jar do?

In which scenarios would it be safe to remove the argument?

--David

[Brad Wood]

It allows for Lucee to rewrite the bytecode inside a class file on the fly instead of just creating new class files when a file changes.  This keeps from using up class files in permgen.  The java agent is optional though and Lucee will work fine without it.  I've actually hit a couple bugs recently related to compilation errors and performance issues in some scenarios with the java agent so I remove it sometimes anyway.  With Java 8, there's not a concern with running out of permgen space anyway.  Micha can chime in if he wants with additional info about that jar.

Thanks!

~Brad

ColdBox Platform Evangelist

Ortus Solutions, Corp 

E-mail: br...@coldbox.org

ColdBox Platform: http://www.coldbox.org 

Blog: http://www.codersrevolution.com

--
Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html
---

You received this message because you are subscribed to a topic in the Google Groups "Lucee" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lucee/56WHZlWc9rI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lucee+un...@googlegroups.com.

To post to this group, send email to lu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/b9b669bf-b44c-4b78-829d-aa317bbc8fdb%40googlegroups.com.