lucee-inst.jar slows down startup

[Mischa Sameli]

We have an issue with lucee-inst.jar. Without having it enabled our web-contexts (about 20) start up in between 1 to 3 minutes. Having lucee-inst.jar enabled, we can wait for more than 15 minutes (high CPU usage all the time) and usually they won't start up at all.
Some words to our environment:
- Java 1.8
- Nginx & Tomcat 8
- lucee 4.5.2
The application itself uses transferOrm, so we generate a lot of classes on the fly.

The reason why I'd like to enable it, is that we have regular crashes because of low free perm gen space.

Any ideas why lucee-inst.jar slows down startup?
Cheers
Mischa

[Andrew Dixon]

Hi Mischa,

Don't have this particular issue with lucee-inst.jar however take a look at this thread about Lucee startup time:

https://groups.google.com/forum/#!topic/lucee/Z0MqvS7z9gw

There are some suggestions in there that really do make it much much faster to startup.

Kind regards,

Andrew

about.me - mso - Lucee Association Member

--
See Lucee at CFCamp Oct 22 & 23 2015 @ Munich Airport, Germany - Get your ticket NOW - http://www.cfcamp.org/
---
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/cd4215ce-df1a-4649-9e53-02d3913719ca%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mischa Sameli]

Hi Andrew
Thanks for the optimizations you've mentioned. They are already applied. And the issue in my eyes is not related to startup speed of tomcat. 1 to 2 minutes for 20 contexts is fair. But the application itself takes hell long to initialize. Sometimes it takes 15 minutes or more and it's not granted that there will be a response.

As far as I know, lucee-inst is used to get smarter class compilation. So classes are overwritten in jvm instead of created once again. And this should help to reduce memory usage and gc work.
Well, I'm afraid that transerOrm or coldSpring which is used in those 20 installations are generating too many new classes on startup.
I don't think it's hardware related, we use SSD and 64GB RAM, not running in a VM.

Any more ideas or experiences?

[Chris Blackwell]

As you say, lucee-inst.jar agent is used for managing recompilation of cfml templates, so if your sites templates/cfcs don't change between restarts, is it doing anything useful for you?

fwiw, we run 4.5.2 without lucee-inst, the app uses coldspring2 and is very component heavy and performance is fine, but our code does not change between restarts and we run one app per jvm.

Chris

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/e92c9281-d5d2-4316-884c-94b58e4aea2d%40googlegroups.com.

[Michael Offner]

Could be related to your perm gen space problem. The Lucee-Inst.jar adds a Java agent to the environment that scans every class loaded, so if you have a lot of classes this could be the problem. How big is your cfclasses folders?

There is a (experimental) way to reduce the size of classes by extracting text fragments out of the classes with help of a simple setting. I will look it up and let you know.

Micha

--

[Mischa Sameli]

Hi Micha
There are about 2000 classes (25 mb) present for each context. But I don't know whether this is huge, normal or just not worth mentioning...
Thanks for your precious time.
Mischa

Am Montag, 19. Oktober 2015 09:21:51 UTC+2 schrieb Micha:

Could be related to your perm gen space problem. The Lucee-Inst.jar adds a Java agent to the environment that scans every class loaded, so if you have a lot of classes this could be the problem. How big is your cfclasses folders?

There is a (experimental) way to reduce the size of classes by extracting text fragments out of the classes with help of a simple setting. I will look it up and let you know.

Micha

Am Sonntag, 18. Oktober 2015 schrieb Mischa Sameli :

We have an issue with lucee-inst.jar. Without having it enabled our web-contexts (about 20) start up in between 1 to 3 minutes. Having lucee-inst.jar enabled, we can wait for more than 15 minutes (high CPU usage all the time) and usually they won't start up at all.
Some words to our environment:
- Java 1.8
- Nginx & Tomcat 8
- lucee 4.5.2
The application itself uses transferOrm, so we generate a lot of classes on the fly.

The reason why I'd like to enable it, is that we have regular crashes because of low free perm gen space.

Any ideas why lucee-inst.jar slows down startup?
Cheers
Mischa

--
See Lucee at CFCamp Oct 22 & 23 2015 @ Munich Airport, Germany - Get your ticket NOW - http://www.cfcamp.org/
---
You received this message because you are subscribed to the Google Groups "Lucee" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lucee+unsubscribe@googlegroups.com.

[Mischa Sameli]

Hi Chris
We're still on ColdSprint 1.x because of stability issues with coldSpring 2. But I don't think this makes the difference. Do you know how many classes are present in your cfclasses folder?
Maybe 1 app per jvm could be a way to go for us. But that means a lot of configuration.
Our application code doesn't change as well between restarts. But we cleanup transferOrm generated files. And these classes are the only ones that are recreated - in theory.
BR
Mischa

[Phil]

Before lucee/railo provided this mechanism, we were cleaning up classes using these java tuning parameters:

                      -XX:+UseConcMarkSweepGC
                      -XX:+CMSClassUnloadingEnabled
                      -XX:+ExplicitGCInvokesConcurrentAndUnloadsClasses

Phil

[Mischa Sameli]

Hi Phil

We do the same, which means we still have these settings - maybe they don't work along with lucee-inst.jar.

And we didn't use the third param: -XX:+ExplicitGCInvokesConcurrentAndUnloadsClasses

Would you recommend it (yes I know: all configurations are different)?

BR

Mischa

--

See Lucee at CFCamp Oct 22 & 23 2015 @ Munich Airport, Germany - Get your ticket NOW - http://www.cfcamp.org/
---

You received this message because you are subscribed to a topic in the Google Groups "Lucee" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lucee/AyNdl8qNgKk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lucee+un...@googlegroups.com.

To post to this group, send email to lu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/32548414-fba0-4b8c-8895-de72d58e4056%40googlegroups.com.

[Phil]

Maybe.. I seem to recall that railo might actually call an explicit GC itself in certain scenarios so it's worth having it clean up classes at that time.  My memory is a bit hazy on this though as it was a while ago.

Phil

[Terry Whitney]

What Operating System?

I ask as there are several known issues depending on OS & processor type with Java & Tomcat.

[Mischa Sameli]

Terry
we use centOS7 64bit

[Michael Heel]

Hi Misscha,

 

we have a JAVA Setting especially for centos:

 

export CATALINA_OPTS="$CATALINA_OPTS -Djava.security.egd=file:/dev/./urandom"

 

 

 

#! /bin/sh

# ==================================================================

 

CATALINA_HOME=/opt/tomcat

 

export CATALINA_OPTS="$CATALINA_OPTS -server"

export CATALINA_OPTS="$CATALINA_OPTS -Xms2048m"

export CATALINA_OPTS="$CATALINA_OPTS -Xmx2048m"

export CATALINA_OPTS="$CATALINA_OPTS -Xss256k"

export CATALINA_OPTS="$CATALINA_OPTS -XX:+UseG1GC"

export CATALINA_OPTS="$CATALINA_OPTS -XX:+ParallelRefProcEnabled"

export CATALINA_OPTS="$CATALINA_OPTS -XX:MaxGCPauseMillis=200"

export CATALINA_OPTS="$CATALINA_OPTS -XX:InitiatingHeapOccupancyPercent=70"

export CATALINA_OPTS="$CATALINA_OPTS -XX:G1ReservePercent=15"

export CATALINA_OPTS="$CATALINA_OPTS -XX:ParallelGCThreads=20"

export CATALINA_OPTS="$CATALINA_OPTS -XX:ConcGCThreads=5"

export CATALINA_OPTS="$CATALINA_OPTS -XX:+AggressiveOpts"

export CATALINA_OPTS="$CATALINA_OPTS -Djava.security.egd=file:/dev/./urandom"

export CATALINA_OPTS="$CATALINA_OPTS -javaagent:/opt/tomcat/lucee/lucee-inst.jar"

export CATALINA_OPTS="$CATALINA_OPTS -Doracle.jdbc.J2EE13Compliant=true"

 

# Check for application specific parameters at startup

if [ -r "$CATALINA_BASE/bin/appenv.sh" ]; then

  . "$CATALINA_BASE/bin/appenv.sh"

fi

 

echo "Using CATALINA_OPTS:"

for arg in $CATALINA_OPTS

do

    echo ">> " $arg

done

echo ""

 

echo "Using JAVA_OPTS:"

for arg in $JAVA_OPTS

do

    echo ">> " $arg

done

echo "_______________________________________________"

echo ""

 

 

Michael Heel

 

CONTENS Software GmbH
Content Management intuitive, reliable and flexible

http://www.contens.com

--

See Lucee at CFCamp Oct 22 & 23 2015 @ Munich Airport, Germany - Get your ticket NOW - http://www.cfcamp.org/
---

You received this message because you are subscribed to the Google Groups "Lucee" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.

To post to this group, send email to lu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/e587fb5e-3229-4b54-acae-eadecb0d25b9%40googlegroups.com.

[Terry Whitney]

Check for the existance of /dev/urandom

if it does not exist, then create it

 mknod -m 644 /dev/urandom c 1 9

 chown root:root /dev/random /dev/urandom

then add the following

vi /etc/rc.local

add the following to the file:

#/bin/bash!

###

      if [ -f /var/random-seed ]; then
                cat /var/random-seed >/dev/urandom
        fi
        dd if=/dev/urandom of=/var/random-seed count=1

####

Then save the file;

Run the command:

chmod a+x /etc/rc.local

reboot the server

On Sunday, October 1

8, 2015 at 6:21:45 AM UTC-4, Mischa Sameli wrote:

[Terry Whitney]

One more thing, in tomcat

// File: conf/context.xml, 

You can  change the scanning method. Granted you will want to test this before you just throw it out to production,

<JarScanner scanClassPath="false" />    That will keep Tomcat from trying to scan every freaking file on init.