Re: [railo] 503 every day

[Michael Offner]

do you can see anything in the log files produced by Railo?

Install and enable the Scope and Memory Monitor Extensions, perhaps you can see something there.

micha

2012/11/14 Keith Reiter <ke...@reiter-family.com>

I have railo 4 set up on a Amazon Micro.  No performance issue or any other issues for that matter.  CFML works great after a Railo restart but then dies within 24 hours every time.  All HTML and PHP file run fine but CFML pages give 503 errors.  Even the admin is unreachable.  I ssh into the box and restart Railo and all is well for another 24 hours.

My question is where do I start looking to find out what is breaking it.  This is on a personal server that gets maybe 20 visits a week so it's not a load issue.

http://cl.ly/image/0E101K1G3x20

I am a long time CFML programmer that has tried to move to Railo or OBD several times and always ran into frustrations with Tomcat or something like that.  I really REALLY want to use and like Railo so any help would be appreciated.

I am pretty familiar with Apache.  I am very familiar with Linux.  I am not familiar with Railo or Tomcat.

--

/micha

Michael Offner CTO Railo Technologies GmbH

[Keith Reiter]

I looked in all the logs by Railo and only see an old reference to "table doesn't exist" (I moved this code from windows and had a case issues..it's fixed). I restarted Railo this morning and it wouldn't come back. I tried several times. Then I restarted Apache and then Railo and it came back. Thaat leads me to believe that Tomcat is the one failing and taking Railo down with it. That seem resonable? Where would I look in Tomcat for the cause.

-Keith

[Peter Boughton]

How did you install Railo/Tomcat (via installer or other means?)

Have you checked the Tomcat logs (i.e. not just the Railo logs - defaults to /opt/railo/tomcat/logs if you used the installer)?

[Matt Quackenbush]

I suspect Tomcat's memory setting is super low (default is like 64 or maybe 128?) and it's falling over. Where to change that is largely dependent upon your OS and how you installed Tomcat. If you can provide that information, we can help guide you more accurately to the right spot.

If you happen to be on Ubuntu, look in /etc/default/tomcat7 (assuming you've installed Tomcat 7).

[Keith Reiter]

Catalina.out has this:


        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:322)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:450)

Nov 14, 2012 3:39:26 AM org.apache.tomcat.util.digester.Digester fatalError
SEVERE: Parse Fatal Error at line 36 column 35: <Line 36, Column 35>: XML-20190: (Fatal Error) Whitespace required.
org.xml.sax.SAXParseException; lineNumber: 36; columnNumber: 35; <Line 36, Column 35>: XML-20190: (Fatal Error) Whitespace required.
        at oracle.xml.parser.v2.XMLError.flushErrorHandler(XMLError.java:415)
        at oracle.xml.parser.v2.XMLError.flushErrors1(XMLError.java:284)
...
Nov 14, 2012 3:39:26 AM org.apache.catalina.mbeans.GlobalResourcesLifecycleListener createMBeans
SEVERE: Exception processing Global JNDI Resources
javax.naming.NamingException: <Line 36, Column 35>: XML-20190: (Fatal Error) Whitespace required.
...
Several more with same "Whitespace required" on the same line and column.  Doesn't tell me what file I need to look at to add whitespace.

Here are the last entries in catalina.out before the reboot this morning:

SEVERE: Error deploying web application directory /opt/railo/tomcat/webapps/tomcat-manager
java.lang.IllegalArgumentException: addChild:  Child name '/tomcat-manager' is not unique
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:887)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:618)
        at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1100)
        at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1618)
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

Nov 14, 2012 3:40:01 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /opt/railo/tomcat/webapps/tomcat-docs
Nov 14, 2012 3:40:01 AM org.apache.catalina.startup.HostConfig deployDirectory
SEVERE: Error deploying web application directory /opt/railo/tomcat/webapps/tomcat-docs
java.lang.IllegalArgumentException: addChild:  Child name '/tomcat-docs' is not unique
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:887)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:618)
        at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1100)
        at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1618)
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

Nov 14, 2012 3:40:01 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /opt/railo/tomcat/webapps/host-manager
Nov 14, 2012 3:40:01 AM org.apache.catalina.startup.HostConfig deployDirectory
SEVERE: Error deploying web application directory /opt/railo/tomcat/webapps/host-manager
java.lang.IllegalArgumentException: addChild:  Child name '/host-manager' is not unique        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:887)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:618)
        at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1100)
        at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1618)
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

[Keith Reiter]

Been a couple months since I installed and tried a few things so I can't tell you exactly how I installed but it is on Ubuntu and there is no /etc/default/tomcat7.  There is a file in my home directory named railo-4.0.0.013-BETA2-linux-x64-installer.run.  I install apps to /opt and there are /opt/railo and /opt/railo/tomcat directories.

-Keith

[Michael Offner]

tomcat is not able to deploy configured web contexts, you should definitely review your tomcat config.

/micha

2012/11/14 Keith Reiter <ke...@reiter-family.com>

[Keith Reiter]

The CFML processes properly for a few hours. Here is what I added to server.xml:
<Host name="mydomain.com" appBase="webapps">
<Context path="" docBase="/websites/mydomain.com" />
<Alias>foo.mydomain.com</Alias>
</Host>
I don't think I touched anything else.

[Keith Reiter]

I started the services around an hour ago and it just failed.

Here is whats in the catalina.out

Nov 14, 2012 2:17:46 PM org.apache.catalina.util.SessionIdGenerator createSecureRandom
INFO: Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [105] milliseconds.
railo-server-root:/opt/railo/lib/railo-server
/opt/railo/lib/railo-server/patches/4.0.0.013.rc
===================================================================
SERVER CONTEXT
-------------------------------------------------------------------
- config:/opt/railo/lib/railo-server/context
===================================================================

Wed Nov 14 14:17:53 UTC 2012-359 using JRE Date Library
Wed Nov 14 14:18:11 UTC 2012-66 Start CFML Controller
Wed Nov 14 14:18:11 UTC 2012 Loaded Railo Version 4.0.0.013
===================================================================
WEB CONTEXT (0f38c727aa462476f7c8b4774faa15a0)
-------------------------------------------------------------------
- config:/websites/keithreiter.com/WEB-INF/railo
- webroot:/websites/keithreiter.com/
- hash:0f38c727aa462476f7c8b4774faa15a0
- label:0f38c727aa462476f7c8b4774faa15a0
===================================================================

Nov 14, 2012 2:18:18 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /opt/railo/tomcat/webapps/tomcat-manager
===================================================================
WEB CONTEXT (3330203f4c141dbc2f4ce9055dcda4c3)
-------------------------------------------------------------------
- config:/opt/railo/tomcat/webapps/tomcat-manager/WEB-INF/railo
- webroot:/opt/railo/tomcat/webapps/tomcat-manager/
- hash:3330203f4c141dbc2f4ce9055dcda4c3
- label:3330203f4c141dbc2f4ce9055dcda4c3
===================================================================

Nov 14, 2012 2:18:20 PM org.apache.catalina.startup.HostConfig deployDirectory

INFO: Deploying web application directory /opt/railo/tomcat/webapps/tomcat-docs

===================================================================
WEB CONTEXT (a4e3e988f47d91b57537a2a88ded4235)
-------------------------------------------------------------------
- config:/opt/railo/tomcat/webapps/tomcat-docs/WEB-INF/railo
- webroot:/opt/railo/tomcat/webapps/tomcat-docs/
- hash:a4e3e988f47d91b57537a2a88ded4235
- label:a4e3e988f47d91b57537a2a88ded4235
===================================================================

Nov 14, 2012 2:18:20 PM org.apache.catalina.startup.HostConfig deployDirectory

INFO: Deploying web application directory /opt/railo/tomcat/webapps/host-manager

===================================================================
WEB CONTEXT (46c377f39fb108b008e4ed163269dc1e)
-------------------------------------------------------------------
- config:/opt/railo/tomcat/webapps/host-manager/WEB-INF/railo
- webroot:/opt/railo/tomcat/webapps/host-manager/
- hash:46c377f39fb108b008e4ed163269dc1e
- label:46c377f39fb108b008e4ed163269dc1e
===================================================================

Nov 14, 2012 2:18:20 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /opt/railo/tomcat/webapps/tomcat-manager
Nov 14, 2012 2:18:21 PM org.apache.catalina.startup.HostConfig deployDirectory

INFO: Deploying web application directory /opt/railo/tomcat/webapps/tomcat-docs

Nov 14, 2012 2:18:21 PM org.apache.catalina.startup.HostConfig deployDirectory

INFO: Deploying web application directory /opt/railo/tomcat/webapps/host-manager

Nov 14, 2012 2:18:21 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /opt/railo/tomcat/webapps/ROOT
===================================================================
WEB CONTEXT (ead619704c9d33daf2e74476a9164724)
-------------------------------------------------------------------
- config:/opt/railo/tomcat/webapps/ROOT/WEB-INF/railo
- webroot:/opt/railo/tomcat/webapps/ROOT/
- hash:ead619704c9d33daf2e74476a9164724
- label:ead619704c9d33daf2e74476a9164724
===================================================================

Nov 14, 2012 2:18:22 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-8888"]
Nov 14, 2012 2:18:22 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["ajp-bio-8009"]
Nov 14, 2012 2:18:22 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 44712 ms
Nov 14, 2012 2:18:25 PM org.apache.catalina.startup.HostConfig deployDescriptor
INFO: Deploying configuration descriptor /opt/railo/tomcat/conf/Catalina/www.keithreiter.com/ROOT.xml
Nov 14, 2012 2:18:30 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /opt/railo/tomcat/webapps/tomcat-manager
Nov 14, 2012 2:18:30 PM org.apache.catalina.startup.HostConfig deployDirectory

INFO: Deploying web application directory /opt/railo/tomcat/webapps/tomcat-docs

Nov 14, 2012 2:18:30 PM org.apache.catalina.startup.HostConfig deployDirectory

INFO: Deploying web application directory /opt/railo/tomcat/webapps/host-manager

I looked in all the railo logs and saw no new entries. I left the site down in case anyone wants to look (keithreiter.com). The site is PHP but I discuss CFML and the examples are CFML (http://www.keithreiter.com/examples/grid/index.cfm)

[Keith Reiter]

"tomcat is not able to deploy configured web contexts, you should definitely review your tomcat config."

That sounds like it could be the case since I know zero about Tomcat.  It also means zero to me.  I installed using the installer and made the entry at the bottom for my virtual host.  Are there other settings that have to be made to make it stable?  I am more than willing to do digging and grab any info needed.  I am hoping that I don't have to become a Tomcat expert to make this run smooth.  From what I've looked into there is nothing in th logs to indicate why it's failing.  I suppose it could be permissions but it works for a few hours.

If I scrapped what I have and go with Railo 3 will it possibly resolve the issue?

-Keith

[Jordan Michaels]

This looks correct.

Are you deploying any sites in the /opt/railo/tomcat/webapps/ROOT/ directory? If you are, it's possible your contexts are conflicting, and that could be causing the problems you're experiencing. Placing your sites in /websites/mydomain/ and NOT in the tomcat ROOT folder is a good idea.

-Jordan

[Keith Reiter]

I placed no web content anywhere in the /opt folder.  I got in the habit of using /websites about 10 years ago and just never changed.  My /website/mydomain was a mapped folder to a file server so if the machine failed I could turn on another generic configured blade and make a few lines of change in the /etc/fstab and /etc/httpd/httpd.conf (RHEL at the time) and be back online from total hardware meltdown in less than 10 minutes.

-K

[Jordan Michaels]

There is no failure at the end of this log file. You say this came from a crashed Tomcat?

I'm still thinking you're getting conflicting contexts. In your Tomcat server.xml file, comment out this line:

<Valve className="mod_cfml.core" loggingEnabled="false" waitForContext="3" />

Commenting out this line will remove Tomcat's ability to dynamically create contexts that you have not explicitly created with the <Host> entry. This means that you will be required to add additional <Host> entries to your Tomcat server.xml file (at the bottom - like you've done before) for a domain to properly resolve in Tomcat, but it will also give you full control over that process just in case you are running into conflicting contexts.

-Jordan


----- Original Message -----
From: "Keith Reiter" <ke...@reiter-family.com>
To: ra...@googlegroups.com

[Keith Reiter]

Here is what I get now.  Maybe this helps narrow things down. http://www.keithreiter.com/examples/grid/index.cfm

-K

[Jordan Michaels]

Yeah, see how Railo is looking for the file in the
/opt/railo/tomcat/webapps/ROOT/ folder? That means that Tomcat doesn't
know how to resolve the www.keithreiter.com domain, so it's resolving it
to the default host (like Apache does).

You will need to add a <Host> entry to the Tomcat server.xml file now
and tell Tomcat where to find the files for that domain. IE: a new one
of these:

<Host name="[ENTER DOMAIN NAME]" appBase="webapps">
<Context path="" docBase="[ENTER SYSTEM PATH]" />
<Alias>[ENTER DOMAIN ALIAS]</Alias>
</Host>

When that's added correctly, your file should resolve to where it's
supposed to.

The mod_cfml valve was adding these contexts automatically for you, but
if those contexts overlap, then that might be what's causing the errors
you're getting. Adding the contexts manually like this will be a little
more work, but it will give you more control and rule out conflicting
contexts as the problem.

Hope this helps.

Warm Regards,
Jordan Michaels

[Keith Reiter]

Perfect.  I added a second alias for WWW and it's working.  We shall see over time if this resolves the issue.  Thanks everyone for your help.

-K

[Keith Reiter]

I workd all the way to bedtime but when I got up it's 503 again.  Any more suggestions on where to look...maybe which logs to investigate.  Where to start.  I thought maybe Tomcat but I very well may be wrong.  I can say that Apache and PHP are working fine.

-K

[ronnie otts]

I have seen errors like the ones you posted when the host runs out of disk space.

[Keith Reiter]

I just checked to make sure.  Everything looks fine.
http://cl.ly/image/3E433f1t1d2X
Thanks for the suggestion.

[ronnie otts]

man, thanks for reminding me to reinstall cloudapp!

:)

[Jordan Michaels]

If it's a problemw ith Tomcat it would be in the tomcat catalina.out
file - usually in /opt/railo/tomcat/logs/catalina.out. If it's a problem
with the connector it would be in the mod_jk log file, usually
/var/log/httpd/mod_jk.log.

Anything indicative in those logs?

If not, we can swap out the connector from mod_jk (which uses AJP) to
mod_proxy and use mod_proxy_html. It will basically mirror what you
would see on port 8888.

Did I already ask you if port 8888 still works when you get the 503?

Warm Regards,
Jordan Michaels

On 11/16/2012 10:14 AM, Keith Reiter wrote:
> I workd all the way to bedtime but when I got up it's 503 again. Any
> more suggestions on where to look...maybe which logs to investigate.
> Where to start. I thought maybe Tomcat but I very well may be wrong. I
> can say that Apache and PHP are working fine.
>
> -K
>
> On Tuesday, November 13, 2012 9:48:24 PM UTC-6, Keith Reiter wrote:
>
> I have railo 4 set up on a Amazon Micro. No performance issue or
> any other issues for that matter. CFML works great after a Railo
> restart but then dies within 24 hours every time. All HTML and PHP
> file run fine but CFML pages give 503 errors. Even the admin is
> unreachable. I ssh into the box and restart Railo and all is well
> for another 24 hours.
>
> My question is where do I start looking to find out what is breaking
> it. This is on a personal server that gets maybe 20 visits a week
> so it's not a load issue.
>

> http://cl.ly/image/0E101K1G3x20 <http://cl.ly/image/0E101K1G3x20>

[Matt Quackenbush]

I had a similar issue back in January-ish. It was indeed a Tomcat problem, but nothing showed in any of the logs. The issue was that Tomcat's memory was set way too low. In my particular case, Mura requires an absolute bare minimum of 768mb to operate smoothly. I played with Tomcat settings between 256 and 512, to no avail. Upping it to 768 fixed the problem.

HTH

[Denny]

On 11/16/12 1:49 PM, Matt Quackenbush wrote:
> I had a similar issue back in January-ish. It was indeed a Tomcat problem,
> but nothing showed in any of the logs. The issue was that Tomcat's memory
> was set way too low. In my particular case, Mura requires an absolute bare
> minimum of 768mb to operate smoothly. I played with Tomcat settings between
> 256 and 512, to no avail. Upping it to 768 fixed the problem.

Yeah, I'm suspecting it's something like this. JVM tuning type of deal.

There's not a magic bullet for tuning, as each app is different, but
this is what Bruce came up with which solved his Tomcat7 issues:

JAVA_OPTS="-Xms512m -Xmx1424m -XX:MaxPermSize=512m -XX:NewSize=64m -
XX:MaxNewSize=64m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:
+CMSClassUnloadingEnabled";

And I've had good luck with similar settings. Some depends on JVM
version, etc., but it's a good place to look, considering the issues
exhibited. Huh. Guess Matt is more than just a pretty face! ;-)

Bruce Kirkpatrick and a couple of others have been really freaking
awesome and reported back in depth on their trials and tribulations
regarding tuning garbage collection and such-- taking a gander through
the archives might give you some other ideas.

Not saying this *is* the problem, but it's a suspect, surely.

:Denny

--
Railo Technologies: getrailo.com Professional Open Source
Skype: valliantster (505)510.1336 de...@getrailo.com
GnuPG-FP: DDEB 16E1 EF43 DCFD 0AEE 5CD0 964B B7B0 1C22 CB62

[Keith Reiter]

Here is what mod_jk logs look like.  I assume this is bad.

http://cl.ly/image/202V2m2l3J10

Port 8888 giving a connection timeout.

[Matt Quackenbush]

Ayep. That's bad. And a _really_ good indication that Tomcat has just died. Exactly the situation I dealt with for quite awhile. I'm still wagering that Tomcat simply doesn't have enough memory for the app(s) on the server.

[ronnie otts]

can you post your tomcat server.xml ?

On Fri, Nov 16, 2012 at 4:03 PM, Keith Reiter <ke...@reiter-family.com> wrote:

[Jordan Michaels]

Actually it's good. It means mod_jk is working fine, it's just
complaining because it can't connect to Tomcat. It rules out mod_jk as
the problem.

Try checking the memory settings as others have suggested. Denny is my
hero when it comes to JVM tuning, so I'd try his suggested settings
first. If it still doesn't work try bumping up the permgen as Matt
suggested. ;)

Warm Regards,
Jordan Michaels

[Matt Quackenbush]

Actually, it's -Xms/-Xmx that I was referring to. But close enough. ;-)

[Jordan Michaels]

Doh! Thank you for the polite correction.

You mentioned Mura and that triggered "permgen" in my mind, which is
where I've personally encountered the most trouble with it. Totally my
bad. =(

Warm Regards,
Jordan Michaels

> http://cl.ly/image/__202V2m2l3J10 <http://cl.ly/image/202V2m2l3J10>

[Matt Quackenbush]

Heh. Reading back it seems I wasn't as clear as I had thought I was. :D

[Ronnie Otts]

Disregard request for server.XML

Ronnie Otts

[Keith Reiter]

Since I am running on a EC2 micro I only have 600M to work with.  Those setting would throw me into disk IO correct?  I use wordpress to display pages because I'm all perpeller-head and no artist.  There are literally like 10 CFM files on the site.  Is Railo too large to run on 600M ram?  I don't want to start a flame war with this statement but I've run CF9 on a micro without issue.  Am I asking too much out os a micro?

[Matt Quackenbush]

The short answer to your question is, no, Railo is nowhere near "too large" to run on 600M RAM. I know people who are running it on 256. The much, much, much longer answer is, it's all about tuning the JVM. The intricate details there are far beyond my pay grade, but I would highly recommend Railo Support to help you with that if you don't otherwise have the knowledge/expertise to dig into the nitty gritty.

Of course there are also those on this list that are damn good with this sort of thing (at least two of which have already replied to this thread), so don't discount what help you can get here, too. I'm just suggesting Railo Support if you're in one of those "holy hell! i gotta get this done NOW!" kind of situations. :-)

http://www.getrailo.com/index.cfm/services/

[Denny]

No, Railo has a much smaller memory footprint than ACF, so it's surely
not Railo itself.

More likely your garbage collection settings or the JVM version.

And FWIW, there are servlet containers which use less memory (and do
less-- but still enough for Railo in general) than Tomcat -- winstone
and jetty-runner are fantastic, tiny examples of such.

[John Ceci]

Keith,

I had a similiar issue and the problem was Apache was configured to have too many processes running in the apache conf file.  Once I set some caps (which I had to tweak for a bit) everything settled down.  It seemed like regular html content and php wouldn't have an issue, but Railo would become unresponsive when the number of worker processes got to high.  Here is my apache2 conf:

KeepAlive On

MaxKeepAliveRequests 100

KeepAliveTimeout 5

<IfModule mpm_prefork_module>

    StartServers          5

    MinSpareServers       5

    MaxSpareServers      10

    MaxClients           30

    MaxRequestsPerChild   0

</IfModule>

<IfModule mpm_worker_module>

    StartServers          2

    MinSpareThreads      25

    MaxSpareThreads      75

    ThreadLimit          64

    ThreadsPerChild      25

    MaxClients           30

    MaxRequestsPerChild   0

</IfModule>

You will need to play with the settings some for your particular server, I could probably add more workers safely, but everything is running fine so I didn't see the need to push it...

Hope this helps,

John

--
John Ceci

[Keith Reiter]

I was looking at this

JAVA_OPTS="-Xms512m -Xmx1424m -XX:MaxPermSize=512m -XX:NewSize=64m - 
XX:MaxNewSize=64m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX: 
+CMSClassUnloadingEnabled"; 

and thought that setting Xmx1424m might be a problem if I only have 600m ram.  Am I looking at this wrong?

I will be looking into this much more and I can be patient.  This is my personal blog and only gets 20 hits a week so nobody really cares.  The main reason I want to get this working is that I work for an ISP and we get frustrated at times with ACF and we want to offer our customers an alternative.  I love that we can offer more control for our customers in a safe manner.  There are many things I like about Railo but installation and tuning are quite a bit different.  Before you think that "oh dear and he wants to offer this as a product", I am not the one who maintains the servers.  I am just a programmer.  There are many things that I have had to force ACF to do that Railo does out of the box (load balanced sessions).

[Jordan Michaels]

Try this for 600MB RAM:

JAVA_OPTS="-Xms128m -Xmx512m -XX:MaxPermSize=256m -XX:NewSize=64m -

XX:MaxNewSize=64m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:
+CMSClassUnloadingEnabled";

That's still a little bit more then 600Megs total, but I trust you have
some swap? Your site will run slow if you move into it but at least it
won't crash and you'll know you need to allocate more RAM to your VM.

The default options that the Railo installer uses are smaller then the
default options that ACF uses (256MB max heap instead of 512MB), for the
pure and simple reason that Railo doesn't need as much and can run fine
in smaller environments. For example, the Railo install on my local
machine here is only taking 150MB of RAM, and that's with a little bit
of my code in there as well. JVM tuning will be necessary in either
engine though, and now that ACF is using Tomcat as well, tuning them is
actually pretty similar. The main reason to tune an engine would be your
Apps.

I know you mentioned that your app doesn't get much traffic, but traffic
isn't the best indicator about what memory requirements your App will
have. Compiled classes make a big difference, what kind of data is
stored in the various scopes. etc. It all varies greatly per
application, and doesn't always directly relate to how much traffic you get.

If your server crashes again, you can check to see how much memory
you're using/needing by running the "top" command. Once "top" is
running, hit "M" (capitol M) to sort by memory usage. The RES column is
your most important column, as it shows how much resident memory you're
using. The "java" process is your Railo server. If you're running
multiple java processes, you can hit "c" and top will list out the full
commands used to launch each process. Railo will be listed as the
"/opt/railo/jdk/jre/bin/java" command...

Hope this helps.

Warm Regards,
Jordan Michaels

[Keith Reiter]

I made the change to JAVA in the setenv.sh file.  I assume that is where it goes after a bit of searching.

I also checked on swap.  It seams as though the Ubuntu team didn't feel the need for swap space at all in their team configured 12.04.  I had no swap at all.  Granted I've never "seen" the ram usage get close to needing it but I assume there are times that it spikes and I'm not looking.  I set up a 1GB swap.  Hopefully the changes you suggested as well as adding swap will smooth things out.

I am so used to having much more machine to work with that it's a bit odd working with EC2s.

-Keith

[Daryl Banttari]

Keith,

If you're using Tomcat strictly for Railo and used the "embedded railo" setup option (where the railo jar files aren't in the webapps directory, but rather pointed to in the common.loader entry of catalina.properties), then you don't need any of the "default" apps running to use Railo.  Delete everything in the tomcat\webapps directory (or if that makes you nervous, simply move them somewhere else.)  This should reduce the memory needs of Tomcat.

--Daryl

[Keith Reiter]

Sounds good.  Let me see if the changes I made today solve the issue.  I can further tweak it once I get it stable.  I don't want to make multiple changes at once and not know which one worked.

It was my fault for not looking at the Ubu12 setup and seeing no swap.  Even though I hope to noever use it I think it's good to have in times of memory spike.  If I find it's in use a lot then I either need to trim the app or grow the VM.

-K

[Keith Reiter]

Here is the new TOP info after running about 2 hours.

http://cl.ly/image/1V3Y0k3V253O

As you can see it's using a little swap.  Not sure if it used it cause it was there or if it needed it.  I had to wait for a time when java was on the list to get the capture.

I'll see if it's still running in the morning.

[Tom Miller]

Keith,

 

The key here (with micro instance) is the %st of CPU usage – which is “steal time”. If that’s high, you’re being throttled by the micro instance, and everything will slow to a crawl.

 

It may be that when the steal time comes back down, railo may be in an unrecoverable state – even though the throttle has been lifted.

 

This may help explain: http://www.youtube.com/watch?v=EQOmqi_n_ZY

 

I gave up on micro instances for this exact reason…

[Jordan Michaels]

This is what the Installer does, so Keith is doing this already. ;)

-Jordan

----- Original Message -----
From: "Daryl Banttari" <dban...@gmail.com>
To: ra...@googlegroups.com
Sent: Monday, November 19, 2012 8:26:12 AM
Subject: Re: [railo] Re: 503 every day

[Daryl Banttari]

The error quoted earlier:

SEVERE: Error deploying web application directory /opt/railo/tomcat/webapps/tomcat-manager

java.lang.IllegalArgumentException: addChild:  Child name '/tomcat-manager' is not unique

Suggests otherwise.

Could be wrong, but doesn't hurt to check.

--Daryl

[Jordan Michaels]

How does that suggest otherwise?

Maybe I'm just being thick and not following your logic. That's entirely
possible!

Warm Regards,
Jordan Michaels

[Daryl Banttari]

Jordan,

"tomcat-manager" is one of the default webapps that ships with Tomcat.  If that directory has been deleted, I'd expect there to be no further reference to it.  The fact that it's presenting an error with that string in it suggests to me that it still exists.  In particular, I can't find any reference to it in the conf directory of my Tomcat installation, where I deleted that dir.

I added Railo to Tomcat manually, instead of installing Railo with Tomcat, so I'm not entirely sure what that install includes or doesn't.

--Daryl

[Jordan Michaels]

Hey Daryl,

I think I figured out where the confusion is. When I stated that "The
installer does that", I was referring to the common-class installation
method, where you add the Railo libs as a Tomcat common class. I was
*not* referring to deleting the Tomcat webapps. Those are included in
the build that the Installer produces.

So, I follow your logic now, and I apologize for the confusion! I was
not clear in my original response.

I don't think the 503 error the OP was mentioning has anything to do
with the default Tomcat apps, but you're right, doesn't hurt to remove
them as a variable. ;)

Warm Regards,
Jordan Michaels

On 11/19/2012 12:56 PM, Daryl Banttari wrote:
> Jordan,
>
> "tomcat-manager" is one of the default webapps that ships with Tomcat.
> If that directory has been deleted, I'd expect there to be no further
> reference to it. The fact that it's presenting an error with that
> string in it suggests to me that it still exists. In particular, I
> can't find any reference to it in the conf directory of my Tomcat
> installation, where I deleted that dir.
>
> I added Railo to Tomcat manually, instead of installing Railo with
> Tomcat, so I'm not entirely sure what that install includes or doesn't.
>
> --Daryl
>
>
> On Mon, Nov 19, 2012 at 11:44 AM, Jordan Michaels <jor...@viviotech.net
> <mailto:jor...@viviotech.net>> wrote:
>
> How does that suggest otherwise?
>
> Maybe I'm just being thick and not following your logic. That's
> entirely possible!
>
> Warm Regards,
> Jordan Michaels
>
>
> On 11/19/2012 09:16 AM, Daryl Banttari wrote:
>
> The error quoted earlier:
>
> SEVERE: Error deploying web application directory

> /opt/railo/tomcat/webapps/__tomcat-manager
> java.lang.__IllegalArgumentException: addChild: Child name

> '/tomcat-manager' is not unique
>
> Suggests otherwise.
>
> Could be wrong, but doesn't hurt to check.
>
> --Daryl
>
>
>
> On Mon, Nov 19, 2012 at 11:01 AM, Jordan Michaels
> <jor...@viviotech.net <mailto:jor...@viviotech.net>

[Keith Reiter]

I wanted to give an update.  The server ran fine all night.  I did change 2 things at once so I need to go back and see what was the cure.  I changed the JAVA settings as suggested AND added swap.  Gut tells me that the lack of swap was the "real" issue with the settings that were there.  I could probably get by with no swap with the right tweaks but I have no issue with swap as long as hits to it are rare and settle quickly.  I'm not running a fortune 500 company here.  Took about 5 hours for the swap usage to hit 60mb and it stopped and stayed there.  I would like to tweak it to not use it all the time.

Attack plan:
1. Remove unused features of Tomcat (as suggested above)
2. Slowly tweak Tomcat memory settings to not need swap if item 1 wasn't enough
3. Increase VM to have more memory (will avoid cause I'm cheap)

I want to thank everyone for the help with this.  I am really a noob to Tomcat.  I expected slowdowns if memory was tight and not total failure but having no swap makes sense now.  I will leave the swap in place cause it can't really hurt but I'll try not to use it.

-K

[Matt Quackenbush]

Good stuff, Keith. Thanks for reporting back.