Production server goes down without any logs

[hjhaz...@gmail.com]

My chicagoboss server randomly goes down without any logs to speak of.

./init.sh attach  

Connecting production system...

Erlang/OTP 17 [erts-6.0] [source] [64-bit] [async-threads:10] [kernel-poll:false]

*** ERROR: Shell process terminated! (^G to start new job) ***

crash.log

2015-03-23 17:57:49 =ERROR REPORT====

HEAD /details/6439/default [goodride_web] 500 0ms

last successfull request served

{"@timestamp":"2015-03-23T18:02:24.748Z","message":"GET / [goodride_web] 200 0ms","level":"INFO","level_as_int":64,"destinations":[],"pid":"<0.746.0>"}

Mar 23 18:02:36 sanmarino kernel: [519802.898389] beam invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0

Mar 23 18:02:36 sanmarino kernel: [519802.898392] beam cpuset=/ mems_allowed=0

Mar 23 18:02:36 sanmarino kernel: [519802.898394] CPU: 0 PID: 13676 Comm: beam Not tainted 3.13.0-36-generic #63-Ubuntu

Mar 23 18:02:36 sanmarino kernel: [519802.898395] Hardware name: Xen HVM domU, BIOS 4.2.amazon 12/03/2014

Mar 23 18:02:36 sanmarino kernel: [519802.898397]  0000000000000000 ffff88007a7ffa80 ffffffff8171e569 ffff88007a501800

Mar 23 18:02:36 sanmarino kernel: [519802.898399]  ffff88007a7ffb08 ffffffff81718e24 0000000000000000 0000000000000000

Mar 23 18:02:36 sanmarino kernel: [519802.898400]  0000000000000000 0000000000000000 0000000000000000 0000000000000000

Mar 23 18:02:36 sanmarino kernel: [519802.898402] Call Trace:

Mar 23 18:02:36 sanmarino kernel: [519802.898407]  [<ffffffff8171e569>] dump_stack+0x45/0x56

Mar 23 18:02:36 sanmarino kernel: [519802.898410]  [<ffffffff81718e24>] dump_header+0x7f/0x1f1

Mar 23 18:02:36 sanmarino kernel: [519802.898413]  [<ffffffff81151ebe>] oom_kill_process+0x1ce/0x330

Mar 23 18:02:36 sanmarino kernel: [519802.898416]  [<ffffffff812d4815>] ? security_capable_noaudit+0x15/0x20

Mar 23 18:02:36 sanmarino kernel: [519802.898418]  [<ffffffff811525f4>] out_of_memory+0x414/0x450

Mar 23 18:02:36 sanmarino kernel: [519802.898420]  [<ffffffff8115896c>] __alloc_pages_nodemask+0xa5c/0xb80

Mar 23 18:02:36 sanmarino kernel: [519802.898423]  [<ffffffff8119903a>] alloc_pages_vma+0x9a/0x140

Mar 23 18:02:36 sanmarino kernel: [519802.898426]  [<ffffffff81179983>] handle_mm_fault+0xb23/0xf00

Mar 23 18:02:36 sanmarino kernel: [519802.898428]  [<ffffffff8172a524>] __do_page_fault+0x184/0x560

Mar 23 18:02:36 sanmarino kernel: [519802.898431]  [<ffffffff8120581f>] ? ep_poll+0x30f/0x340

Mar 23 18:02:36 sanmarino kernel: [519802.898433]  [<ffffffff8172a91a>] do_page_fault+0x1a/0x70

Mar 23 18:02:36 sanmarino kernel: [519802.898434]  [<ffffffff81726d88>] page_fault+0x28/0x30

Mar 23 18:02:36 sanmarino kernel: [519802.898435] Mem-Info:

Mar 23 18:02:36 sanmarino kernel: [519802.898445] Node 0 DMA per-cpu:

Mar 23 18:02:36 sanmarino kernel: [519802.898447] CPU    0: hi:    0, btch:   1 usd:   0

Mar 23 18:02:36 sanmarino kernel: [519802.898447] Node 0 DMA32 per-cpu:

Mar 23 18:02:36 sanmarino kernel: [519802.898448] CPU    0: hi:  186, btch:  31 usd:  24

Mar 23 18:02:36 sanmarino kernel: [519802.898451] active_anon:485508 inactive_anon:70 isolated_anon:0

Mar 23 18:02:36 sanmarino kernel: [519802.898451]  active_file:11 inactive_file:30 isolated_file:0

Mar 23 18:02:36 sanmarino kernel: [519802.898451]  unevictable:0 dirty:4 writeback:0 unstable:0

Mar 23 18:02:36 sanmarino kernel: [519802.898451]  free:13259 slab_reclaimable:2464 slab_unreclaimable:2977

Mar 23 18:02:36 sanmarino kernel: [519802.898451]  mapped:10 shmem:91 pagetables:2335 bounce:0

Mar 23 18:02:36 sanmarino kernel: [519802.898451]  free_cma:0

Mar 23 18:02:36 sanmarino kernel: [519802.898453] Node 0 DMA free:8272kB min:348kB low:432kB high:520kB active_anon:7184kB inactive_anon:0kB active_file:0kB inactive_file:4kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15988kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:56kB slab_unreclaimable:40kB kernel_stack:0kB pagetables:332kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:9 all_unreclaimable? yes

Mar 23 18:02:36 sanmarino kernel: [519802.898456] lowmem_reserve[]: 0 1982 1982 1982

Mar 23 18:02:36 sanmarino kernel: [519802.908526] Out of memory: Kill process 13262 (beam) score 294 or sacrifice child

Mar 23 18:02:36 sanmarino kernel: [519802.912458] Killed process 13262 (beam) total-vm:1309724kB, anon-rss:601996kB, file-rss:0kB

Is there any way for me to restart my server when this occurs

[Eric Cestari]

Your monitoring should show that the server ran out of memory, and the kernel killed the VM process.

Best,

Eric

--
You received this message because you are subscribed to the Google Groups "ChicagoBoss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chicagoboss...@googlegroups.com.
Visit this group at http://groups.google.com/group/chicagoboss.
To view this discussion on the web visit https://groups.google.com/d/msgid/chicagoboss/a9616bf5-4553-4dd9-8eab-61b1b5b3a18d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.