[2.0] Play server dies after some time

[chiappone]

I am running the play 2.0 in production mode on an ubuntu ec2 instance.  The server seems to just die over time for no reason, and nothing is in the logs.  The server has barely any traffic so I can't see it being do to load.  Any ideas?

[Guillaume Bort]

Can you describe which components your application is using?

> --
> You received this message because you are subscribed to the Google Groups
> "play-framework" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/play-framework/-/483Ph5rCVLsJ.
> To post to this group, send email to play-fr...@googlegroups.com.
> To unsubscribe from this group, send email to
> play-framewor...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/play-framework?hl=en.

--
Guillaume Bort

[GrailsDeveloper]

Do you have any information about the memory at the crash? In 1.2.4
there was a similar bug see
http://stackoverflow.com/questions/8697450/play-framework-application-crashes-on-linux/8702102#8702102.
Niels

[chiappone]

The application is barely doing any thing but hosting a single html page, the controller is in scala and only renders the index html.  Memory usage is minimal.

[GrailsDeveloper]

Well a JVM doesn't simply die. There are always some information on
the console or hs_err_pid*.log. Without such information it's
difficult to help.

Niels

[Thomas Alisi]

hey,

I have the same problem, running play 2.0.3 on ubuntu 10.04 LTS running on ec2.

the last message I have in the application.log is:

[INFO] - from play in Thread-6 

Shutdown application default Akka system.

which I think is the last execution, but cannot be 100% sure. I don't have any hs_err_pid*.log in the folder where dist was running, should I find it somewhere else?

pretty scary having a "production" server that dies with no clues...

[Thomas Alisi]

hey guys,

it happened again: on a production server (ubuntu 10.04 lts on ec2) after 2 days of happy processing, the server simply went down, no pid log whatsoever. 

it's really scary and can be a showstopper for our implementation using play. 

I feel it's something related to memory usage, but cannot find logs on the server, so any further suggestions to debug the application are much appreciated

please let me know

cheers

thomas

On Friday, August 17, 2012 5:29:33 PM UTC+1, Alex Povar wrote:

Hi, Thomas

this happens to me too when I use production build with in-memory H2 database. There was just a few rows in one table, but it breaks down whole application.

пятница, 17 августа 2012 г., 17:55:33 UTC+7 пользователь Thomas Alisi написал:

[Sadache Aldrobi]

Can you try to throw a lot of requests at your app and see how memory behaves, is everything correctly GCed?

--

You received this message because you are subscribed to the Google Groups "play-framework" group.

To view this discussion on the web visit https://groups.google.com/d/msg/play-framework/-/eTcCWbB5cfcJ.

To post to this group, send email to play-fr...@googlegroups.com.
To unsubscribe from this group, send email to play-framewor...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/play-framework?hl=en.

--
www.sadekdrobi.com
ʎdoɹʇuǝ

[Thomas Alisi]

you mean manually or using a profiler? if so, do you have anything to suggest?

I'm not interested in checking loads of request that executes in a short amount of time (e.g. 100k+ requests each providing results in mseconds) but the other way round (e.g. 1000 requests returning a result in seconds), hence the reason why I implemented the akka.future function and running production with -Xmx1500m

checking manually with "top" on a normal cycle of operations for few minutes, I see the CPU constantly around 80% and memory 15%, so I don't think there are memory leaks as memory usage doesn't grow substantially unless it's processing really big chunks of data

other suggestions?

cheers

Thomas M. Alisi, PhD
http://www.linkedin.com/in/thomasalisi

[opensource21]

Normally you always get information if a JVM dies. hserr.pid-files if something really bad happens in native code or an exception on the console. So look carefully for such information. This is the reason why you never should catch Throwables without make sure that they are logged. I found this issue today  https://play.lighthouseapp.com/projects/82401/tickets/663 which could be the reason why you don't see anything.  

On Solaris there exists tools which allows to observe a process, so that you can see if the process get a kill -5 or something like that. Don't know if you can find something similar for ubuntu. 

At last you can observe the whole play-process and track the memory. This decrease the performance of course but without information it's difficult to track it's down, specially in such a complex stack like play. If you can't use a profiler you can log regulary the memory information you can get directly from the VM.

Niels

[Tom Carchrae]

On Sun, Aug 19, 2012 at 6:07 AM, Thomas Alisi <thoma...@gmail.com> wrote:
> you mean manually or using a profiler? if so, do you have anything to
> suggest?

http://httpd.apache.org/docs/2.2/programs/ab.html

[Tom Carchrae]

and http://visualvm.java.net/ - which is part of the standard oracle jdk.

[Thomas Alisi]

hey niels

thanks for your note. solaris... is it still in use? :) just joking! I double checked my code, and I never ever catch throwables (reason is that I am using quite a lot of other libraries and want to keep fine grain control of what's happening and who's messing up things). 

I am doing the following:

- adding a few loggers around to see if it's possible to have a better understanding (reading memory from the VM)
- redirecting standard err + output to logfile

- increased cpu + memory on the virtual machine (although I cannot scale too much, I have 4gb + 2cpu now)

I'll make it run for a while with this config and see if I can get more insights.

in the meanwhile, thank you so much for your help

cheers

thomas

To view this discussion on the web visit https://groups.google.com/d/msg/play-framework/-/TS85_JHpEqYJ.

To post to this group, send email to play-fr...@googlegroups.com.
To unsubscribe from this group, send email to play-framewor...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/play-framework?hl=en.

[opensource21]

Don't forget to write your own  onError method on GlobalSettings as a work around for    https://play.lighthouseapp.com/projects/82401/tickets/663 

And if you really want to find the reason I would recommend to use a load-balancer in front of 2 play-server. Give one of them less memory so that this server dies first.

Niels

[Sadache Aldrobi]

On Mon, Aug 20, 2012 at 1:33 PM, Thomas Alisi <thoma...@gmail.com> wrote:

hey niels

thanks for your note. solaris... is it still in use? :) just joking! I double checked my code, and I never ever catch throwables (reason is that I am using quite a lot of other libraries and want to keep fine grain control of what's happening and who's messing up things). 

I am doing the following:

- adding a few loggers around to see if it's possible to have a better understanding (reading memory from the VM)
- redirecting standard err + output to logfile

- increased cpu + memory on the virtual machine (although I cannot scale too much, I have 4gb + 2cpu now)

I would limit memory to see it crashing faster in case the issue is memory

--
www.sadekdrobi.com
ʎdoɹʇuǝ

[Thomas Alisi]

hey chaps,

thank you so much for your help. quick update from this side: thanks to memory tracing and output logging I found that a hungry function is slowly but constantly consuming extra memory (it is a static method for entity extraction, saving documents in memory and never unloading them). 

I will need to see how to redesign the function so it can live in a web server lifecycle without pulling everything else down, but I also have a question regarding garbage collection in play: would you guys recommend to explicitly call the gc sometimes for housekeeping? if so, what would be the best place to do it?

thanks in advance

thomas

[Sadache Aldrobi]

On Tue, Aug 21, 2012 at 4:21 PM, Thomas Alisi <thoma...@gmail.com> wrote:

[...], but I also have a question regarding garbage collection in play: would you guys recommend to explicitly call the gc sometimes for housekeeping? if so, what would be the best place to do it?

Never.

--
www.sadekdrobi.com
ʎdoɹʇuǝ

[Thomas Alisi]

hey guys

sorry to bother you again with this problem, but I fixed the memory leak, still the server goes down, this time for no apparent reason and without giving any exception or suspicious use of memory.

after 3h of happy processing, the jvm was still using almost the same amount of memory and the server just stopped, I can't see any exception in my log, nor any hs_err_ file in my folder. I only have one last message:

[[37minfo[0m] play - Shutdown application default Akka system.

I launched the server with command:

./start -Xmx1536m -Xms512m &> server.log &

any suggestion deeply appreciated

thanks!

[Tom Carchrae]

What version of the JVM are you using? If you are using OpenJDK, try
switching to the Oracle JDK.

If it is a default Ubuntu install, it is probably using OpenJDK.

> https://groups.google.com/d/msg/play-framework/-/dDhptPptKbEJ.

[Thomas Alisi]

nope, it's Sun Java6 on Ubuntu 10.04.4 LTS: Java(TM) SE Runtime Environment (build 1.6.0_26-b03)

btw, I've created my own bash script to run the server, instead of the default "exec java" shipped in the .zip created by play dist. for the time being it has been running for 5h50m, which is my personal record... uhm, a bit confused but running, so far so good.

here's the script, hope this helps for others who share the same pain:

#!/bin/bash

#

BASE_DIR="/my/home/dir/server"

cd $BASE_DIR

# without slashes as it's used for java execution

PLAY_DIR="stamat-web-1.0-SNAPSHOT"

LOG_FILE=$BASE_DIR/server.log

PID_FILE=$BASE_DIR/$PLAY_DIR/RUNNING_PID

if [ -f "$PID_FILE" ]

then

echo "process running > $(cat $PID_FILE) - $PID_FILE"

else

MESSAGE="$(date) - restarting play!"

echo $MESSAGE

java -Xmx1536m -Xms512m -cp "$PLAY_DIR/lib/*" -Dconfig.file=$PLAY_DIR/staging.conf play.core.server.NettyServer $PLAY_DIR &> $LOG_FILE &

fi

[Manuel Bernhardt]

Hi,

> java -Xmx1536m -Xms512m -cp "$PLAY_DIR/lib/*"
> -Dconfig.file=$PLAY_DIR/staging.conf play.core.server.NettyServer $PLAY_DIR
> &> $LOG_FILE &

I think you might need to give it a bit more permgen, for example with
-XX:MaxPermSize=128M

I remember some linux boxes dying without leaving a trace when the JVM
ran out of PermGen space.

Manuel

[Nilanjan]

Have looked into java -XX:ErrorFile=<path to error flie> flag? This flag is suppose to log fatal errors for Oracle JVM

Nilanjan

[Thomas Alisi]

hey chaps, thank you for all the support,

I'll give a go to the parameters you have suggested, but I have good news: running my own script instead of the one packed with the distro, the server has been up and running with 20hrs+ without a hitch. I am not sure why, but I swear it's the only thing I changed, maybe ubuntu doesn't like "exec" when a script is declared as /env sh? sounds weird but these are the facts.

all the best, hope this thread is useful to someone else

cheers

thomas

To view this discussion on the web visit https://groups.google.com/d/msg/play-framework/-/rnubzfVljPYJ.

To post to this group, send email to play-fr...@googlegroups.com.
To unsubscribe from this group, send email to play-framewor...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/play-framework?hl=en.

[opensource21]

As far as I know in Ubuntu is the default sh  the dash . So if you change your skript to use #!/bin/sh instead of #!/bin/bash I would assume you will get the problem back. In this case you should report it to Debian and Ubuntu. Would be interesting to know because in this case I think it's definitely not a problem from play.

Can you check it?

Niels

[Thomas Alisi]

I will as soon as I can, I have this server running which is waaaaaaay late with its duties, but as soon as it's finished I'll give it a whirl and let you know

To view this discussion on the web visit https://groups.google.com/d/msg/play-framework/-/nKgCGY6L9scJ.

To post to this group, send email to play-fr...@googlegroups.com.
To unsubscribe from this group, send email to play-framewor...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/play-framework?hl=en.

[Thomas Alisi]

hi guys,

apologies if it took so long but my servers were running late with a few crucial tasks. 

I can confirm what mentioned above in this thread: play has been running nicely with my own launch script (you can find it a few messages above in this same thread) while it randomly crashes if I execute the start script packed in the play distro. I have a 282MB log file which I should dig to get more insights, but honestly I cannot allocate time on this, since it's now happily running without flaws. 

I'm on ubuntu 10.04 lts with sun jdk 1.6, happy to give more details about server configuration should you need it

cheers, thank you so much for all your support

thomas

To post to this group, send email to play-framework@googlegroups.com.
To unsubscribe from this group, send email to play-framework+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/play-framework?hl=en.

[James L.]

I haven't tried a production mode and could think Linux kernel may caused the crash, would you might test on Debian or FreeBSD distro to confirm the crash is producible?

[Thomas Alisi]

Hey

Unfortunately it's not possible for me, we're running servers on aws using our own preconfigured images based on Ubuntu and honestly I cannot spare resources for this test, apologies.

--

You received this message because you are subscribed to the Google Groups "play-framework" group.

To view this discussion on the web visit https://groups.google.com/d/msg/play-framework/-/hWIB-Jc30-8J.

[Alex Povar]

Sometimes it dies without stacktrace and any other signals because of openVZ issues. If there is openVZ in your virtualization box, try to ask google how to solve that problem. There are a lot things to do, but switching hosting is far more simpler.

суббота, 28 января 2012 г., 23:46:45 UTC+7 пользователь chiappone написал:

[James L.]

openVZ with burstable memory is bad.

On Sunday, 29 January 2012 00:46:45 UTC+8, chiappone wrote:

[Nico]

@Thomas I had the same problem with a AWS micro instance; problem was that micro instances has 630Mb of RAM and no swap, so the kernel just kills the process if it uses too much memory (I saw that using the "dmesg" linux command).