Application silently shuts down by itself after running for some hours

[JokkeB]

I have an application which I run with "lein run". The main looks something like this

(defn -main []
 (log/info "start")
 (log/info "channel returned" (async/<!! (start-server (socket-server 9121) (websocket-server 9122))))
 (log/info "quit"))

start-server returns a channel created with async/reduce which shouldn't ever produce a value. I see the "start" logged, the application runs for a day or two (not sure if the time is always the same) and then it closes without any errors or without logging "channel returned" or "quit". 

If there is an error in my code and the program quits I should see the "quit" message. If there is a memory leak or something I would assume I'd get an error message. Has anyone experienced anything similar?

Could the reason be running it with lein (I haven't tried a jar)? If so, why?

[Scott Bauer]

Out of curiosity, have you tried catching any and all errors as opposed to expecting to see an error message logged upon the system dying?

While searching for possible causes, I did come across this similar post from a few years ago... not sure if it will help, but it may.

[JokkeB]

Thanks for the reply.

I haven't tried catching any or all errors. Is it a false assumption that an error should be logged if it comes from the main thread? Can an exception in another thread kill the app? Anyways, I now added this piece of code in main, like instructed in https://stuartsierra.com/2015/05/27/clojure-uncaught-exceptions and let's see what happens. Is this what you meant?

  (Thread/setDefaultUncaughtExceptionHandler
   (reify Thread$UncaughtExceptionHandler
     (uncaughtException [_ thread ex]
       (log/error ex "Uncaught exception on" (.getName thread)))))

[JokkeB]

After adding this, I still can't see an exception before the app dies.

[Kevin Corcoran]

On Mon, Mar 6, 2017 at 11:56 PM, JokkeB <jahv...@gmail.com> wrote:

After adding this, I still can't see an exception before the app dies.

I've encountered this before when the Linux "OOM killer" kicks in, which is especially likely if you are running your application on a resource-constrained system (say, a VM with a low RAM allocation) or your application is competing with other programs for memory.

[piast...@gmail.com]

I asked the same question a year ago. The problem was that I was getting an OutOfMemoryError. This is an Error but it is not an Exception. If you catch all Exceptions, you will still not catch the OutOfMemoryError. You have to catch that too. 

[Alex Miller]

If you are getting an OOME there are some JVM flags that can help dump an error file or heap dump, or run arbitrary commands when an error occurs:

-XX:ErrorFile=./hs_err_pid<pid>.log

-XX:+HeapDumpOnOutOfMemoryError 

-XX:HeapDumpPath=./java_pid<pid>.hprof

-XX:OnError="<cmd args>;<cmd args>"

-XX:OnOutOfMemoryError="<cmd args>;<cmd args>"

That might help you determine the cause if it is an error condition.

[Johannes Ahvenniemi]

I'm under the impression that setDefaultExceptionHandler still catches OOM errors. Some googling suggests this too. If not, how should I try to catch it?

I am running the app on a virtual server with 512mb ram. free -m is showing 30mb free. Lack of memory can be the issue. What would be the best way to confirm this if I'm not able to log the error?

Sent from my iPhone

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/zhdcNMC7fQ8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[lawrence...@gmail.com]

To catch OutOfMemoryError s: 

catch(OutOfMemoryError e)

[Mark Nutter]

I see this thread is going on for a while, and I'll say up front that I don't have any helpful insights. I did see an interesting talk at Abstractions, though, about how to use dtrace to get to the bottom of weird problems like this. The video for that event doesn't seem to be online anywhere, but the speaker gave a similar talk at !!Con, and that one is online. If you're still having issues you might want to check this out, in case it gives you an extra tool to use for tracking this down.

https://www.youtube.com/watch?v=1OMX69KOhGg 

Good luck.

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to

clojure+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---

You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscribe@googlegroups.com.

[JokkeB]

Thanks to all the replies. Yes, the reason indeed was linux killing the process. cat /var/log/kern.log verified this. There was no exception logged because there was no exception/error. Now tracking down if I'm simply too low on memory or if the app has a memory leak, but that's a different topic. 

Thanks again!

[Herwig Hochleitner]

2017-03-08 9:00 GMT+01:00 JokkeB <jahv...@gmail.com>:

Now tracking down if I'm simply too low on memory or if the app has a memory leak, but that's a different topic.

Your app may well be fine. The OOM killer removes processes in descending order of memory usage, when RAM runs out. As java processes are usually large, those are amongst the first victims.

This happens even when your app has plenty of free heap, because the java garbage collector reserves a lot of memory before allocating it, and you only get OutOfMemoryExceptions, when the GC hits its limit (set by -Xmx).

To fix the OOM killer issue, simply add more swap space.