Jenkins slaves are killing jobs with a linux kernal memory error

[Sean Last]

Thu Jan 29 15:48:45 2015] Out of memory: Kill process 19422 (java) score 139 or sacrifice child
[Thu Jan 29 18:24:25 2015] Out of memory: Kill process 19422 (java) score 140 or sacrifice child
[Thu Jan 29 20:23:21 2015] Out of memory: Kill process 19422 (java) score 143 or sacrifice child
[Fri Jan 30 20:25:04 2015] Out of memory: Kill process 19422 (java) score 162 or sacrifice child
[Tue Feb 3 19:19:10 2015] Out of memory: Kill process 19422 (java) score 163 or sacrifice child
[Wed Feb 4 14:16:41 2015] Out of memory: Kill process 19422 (java) score 165 or sacrifice child
[Wed Feb 4 16:22:11 2015] Out of memory: Kill process 19422 (java) score 166 or sacrifice child
[Wed Feb 4 19:20:27 2015] Out of memory: Kill process 19422 (java) score 163 or sacrifice child
[Fri Feb 6 14:16:21 2015] Out of memory: Kill process 25783 (java) score 112 or sacrifice child
[Mon Feb 9 06:20:54 2015] Out of memory: Kill process 31175 (java) score 140 or sacrifice child
[Mon Feb 9 08:55:12 2015] Out of memory: Kill process 25783 (java) score 115 or sacrifice child
[Mon Feb 9 16:55:25 2015] Out of memory: Kill process 25783 (java) score 121 or sacrifice child
[Mon Feb 9 16:55:59 2015] Out of memory: Kill process 25783 (java) score 121 or sacrifice child
[Mon Feb 9 17:01:16 2015] Out of memory: Kill process 25783 (java) score 121 or sacrifice child
[Wed Feb 11 10:13:51 2015] Out of memory: Kill process 25783 (java) score 127 or sacrifice child
[Wed Feb 11 10:16:37 2015] Out of memory: Kill process 25783 (java) score 127 or sacrifice child

Should I reduce the number of executors, or how can I solve this?

[Mark Waite]

I think that message usually means you've exhausted virtual memory.  With most systems, if you've exhausted virtual memory, you've already been suffering poor performance due to thrashing of your paging system (swap space).

Reduce the number of executors, or increase the physical memory and possibly the size of the paging file(s).

Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/4915c03a-d298-4059-b410-d14e2a9d981e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks!

Mark Waite

[Sean Last]

Is there a good metric to reduce the number of executors by?

[Mark Waite]

On Tue, Feb 17, 2015 at 11:33 AM, Sean Last <qkt...@gmail.com> wrote:

Is there a good metric to reduce the number of executors by?

My heuristic is that I try to keep the number of executors down to no more than one less than the number of processors on the computer.  However, another heuristic might be to watch the memory use of your jobs and limit the number of executors so that your jobs never exceed the physical memory available on the system.

I'm sure others have different heuristics based on their job types.  My job types tend to be more CPU bound than memory bound.

Mark Waite

 

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/9b95be75-2f69-43d8-8fdc-af0e04a7baf3%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks!

Mark Waite

[Scott Evans]

Sean,

Without knowing your specifics of memory capacity, number of current executors, executor usage, etc., it's hard to give any hard guidelines.  Perhaps try to reduce by 25% the number of executors and see what happens.  There's no magic number as each installation is different.

Scott

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/9b95be75-2f69-43d8-8fdc-af0e04a7baf3%40googlegroups.com.

[Sean Last]

My understanding of jenkins is that all the executors on a slave run in the same JVM?  Is that correct? Or does it launch separate JVMs for each executor?  If I manually set the heap size (Xmx) for each executor, could I know exactly how much RAM was being used?

[Baptiste Mathus]

No executors on a slave run in the same JVM.

From my experience, when possible and for non trivially might builds I'd recommend to run only one executor per slave. This way you won't have difficult to predict instabilities when X ram-greedy builds will run simultaneously on the same slave.
If you can afford using VM, then just spawn smaller ones but more of them.
My 2 cents

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/fdffd9ea-085f-405e-a66e-9aaf0ba36645%40googlegroups.com.