Jobs starting on slaves with no disk space

[Pawel]

Hi,

I have a problem with jobs being started on a slaves that are low on disk
space. I know there is a setting to turn the slaves offline automatically
when disk space falls behind a certain threshold, but it isn't reliable
enough. I think the problem may be with how often the check is performed (as
it can take up to an hour to turn the slave back online automatically after
freeing up enough disk space). The problem in my situation is when I have
big load on my slaves (many jobs and long queue). I have a job running on a
slave X which makes it get low on disk space. After the job finishes the
slaves doesn't get turned off automatically though - new jobs get assigned
to this node and are started, but fail with "no space left on device" error
message just a while later. Jenkins assigns all fitting jobs from the queue
this way, resulting in multiple failures, before node gets turned offline.
Is there any way to prevent this behavior?



--
View this message in context: http://jenkins.361315.n4.nabble.com/Jobs-starting-on-slaves-with-no-disk-space-tp4656054.html
Sent from the Jenkins users mailing list archive at Nabble.com.

[Sami Tikka]

Install post-build-task plugin and use that to clean the workspace at the end of the build.

-- Sami

> --
> You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Pawel]

That's not a solution I can accept - I need those workspaces at least for
some time. They are being wiped out every 24h.
For me it looks like a bug in Jenkins - as problems like that shouldn't
happen.



--
View this message in context: http://jenkins.361315.n4.nabble.com/Jobs-starting-on-slaves-with-no-disk-space-tp4656054p4656972.html

[William Soula]

Create a downstream job that kills the slave by running the following command:
curl -d "offlineMessage=&json=%7B%22offlineMessage%22%3A+%22%22%7D&Submit=Yes" http://[user]:[userApi]@[jenkinsServer]/computer/[slaveName]/doDisconnect

you can leave off [user]:[userApi] if authentication is not necessary. Probably need to change the setting for keeping the slave online.

Will

________________________________________
From: jenkins...@googlegroups.com [jenkins...@googlegroups.com] on behalf of Pawel [pgronk...@gmail.com]
Sent: Monday, February 25, 2013 2:29 AM
To: jenkins...@googlegroups.com
Subject: Re: Jobs starting on slaves with no disk space

--

[Sami Tikka]

I have a couple of jobs also that screw up the slave so badly it has to be taken offline and fixed before it can be used again. I do it using the Groovy plugin and this system groovy script which runs in a build step:

import hudson.model.*
def thr = Thread.currentThread()
def build = thr?.executable
build.getBuiltOn().toComputer().setTemporarilyOffline(true)

The upside is it requires no authentication and does not require the slave to be able to connect to master. These slaves happen to run in an isolated network that blocks connections to the network where the master sits.

-- Sami

[Pawel]

William:

Downstream job won't work, because it will be queued and it will be too late
before it gets executed.

Sami:

Yes, groovy sounds like a good idea.


I have groovy-postbuild plugin, so I will add a check for disk space and I
can turn the slave offline there.



--
View this message in context: http://jenkins.361315.n4.nabble.com/Jobs-starting-on-slaves-with-no-disk-space-tp4656054p4658075.html

[Pawel]

I have expanded the script a bit, so it checks for free disk space:

import hudson.model.*;
import hudson.node_monitors.DiskSpaceMonitor;
import java.math.BigDecimal;

def thr = Thread.currentThread()
def computer = thr?.executable.getBuiltOn().toComputer()
def space = DiskSpaceMonitor.DESCRIPTOR.get(computer);
BigDecimal bd = new BigDecimal(space.getGbLeft());
long value = bd.setScale( 0, BigDecimal.ROUND_HALF_UP ).longValue();
if(value < X) {
computer.setTemporarilyOffline(true);
}

Hopefuly Jenkins gets fixed so things like that are not required.



--
View this message in context: http://jenkins.361315.n4.nabble.com/Jobs-starting-on-slaves-with-no-disk-space-tp4656054p4658461.html