CLI/scripted way to stop scheduling any new builds on a given node?

637 views
Skip to first unread message

Tomasz Chmielewski

unread,
Oct 26, 2017, 5:35:29 AM10/26/17
to Jenkins Users
Is there a CLI/scripted way to stop scheduling any new builds on a given node?

Basically, any builds currently running on a given node should continue to run until they are finished -- and no new builds should be started.

Think of "retiring" a node, and replacing it with a new one -- but allowing any existing jobs to finish gracefully.



Tomasz Chmielewski
https://lxadm.com

Robert Hales

unread,
Oct 26, 2017, 11:21:17 AM10/26/17
to Jenkins Users
In the CLI, use the 'offline-node' command. Another useful command in what it looks like you want to do might be "wait-offline-node".

You could also create a groovy script to do it and run that from the REST API. 

Tomasz Chmielewski

unread,
Oct 26, 2017, 4:55:07 PM10/26/17
to Jenkins Users
Hmm, where do I find "offline-node" command?

root@jenkins:~# dpkg -L jenkins
/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/jenkins
/usr/share/doc/jenkins/changelog.gz
/usr/share/doc/jenkins/copyright
/usr/share/jenkins
/usr/share/jenkins/jenkins.war
/etc
/etc/logrotate.d
/etc/logrotate.d/jenkins
/etc/default
/etc/default/jenkins
/etc/init.d
/etc/init.d/jenkins
/var
/var/cache
/var/cache/jenkins
/var/lib
/var/lib/jenkins
/var/log
/var/log/jenkins

root@jenkins:~# find / -name offline-node

root@jenkins:~#

root@jenkins:~# dpkg -l | grep jenkins
ii  jenkins                          2.73.2       (...)

Robert Hales

unread,
Oct 26, 2017, 4:57:18 PM10/26/17
to Jenkins Users
You have to use the Jenkins CLI. I guess that can be a bit confusing. It isn't a script available to run at the command line. Jenkins has their own CLI. If you google for it, you will find the details pretty easily. 

Tomasz Chmielewski

unread,
Oct 26, 2017, 5:07:03 PM10/26/17
to Jenkins Users

Tomasz Chmielewski

unread,
Oct 26, 2017, 5:18:46 PM10/26/17
to Jenkins Users
Except... it doesn't seem to work.

$ java -jar jenkins-cli.jar -s https://jenkins-url -auth user:pass help offline-node
$ echo $?
255

In nginx log:

10.11.0.8 - user [26/Oct/2017:21:11:51 +0000] "GET / HTTP/1.1" 200 150393 "-" "Java/1.8.0_131"
10.11.0.8 - user [26/Oct/2017:21:11:52 +0000] "GET /crumbIssuer/api/xml/?xpath=concat(//crumbRequestField,\x22:\x22,//crumb) HTTP/1.1" 404 335 "-" "Java/1.8.0_131"
10.11.0.8 - user [26/Oct/2017:21:12:07 +0000] "POST /cli?remoting=false HTTP/1.1" 200 11 "-" "Java/1.8.0_131"
10.11.0.8 - user [26/Oct/2017:21:12:07 +0000] "POST /cli?remoting=false HTTP/1.1" 500 13912 "-" "Java/1.8.0_131"

How do I debug this?

Devin Nusbaum

unread,
Oct 26, 2017, 5:26:39 PM10/26/17
to jenkins...@googlegroups.com
Make sure to follow https://wiki.jenkins.io/display/JENKINS/Running+Jenkins+behind+Nginx if Nginx is configured as a a reverse proxy.

Notably proxy_http_version 1.1; and  proxy_request_buffering off; are required for your version of Jenkins. (If your Jenkins logs at the time you try to connect via CLI have errors that say something to the effect of “Full-duplex channel timeout” then I expect those settings to fix it.)

-- 
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/a19302b1-6ed1-44bb-b65b-28868a64708b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Tomasz Chmielewski

unread,
Oct 26, 2017, 5:43:32 PM10/26/17
to Jenkins Users
nginx vhost is almost the exact copy of the vhost on https://wiki.jenkins.io/display/JENKINS/Jenkins+behind+an+NGinX+reverse+proxy:

upstream jenkins {
    server 127.0.0.1:8080 fail_timeout=0;
}

server {

    listen 80;
    server_name jenkins.my-domain;

    add_header X-Frame-Options SAMEORIGIN;
    include /etc/nginx/release.conf;

    access_log  /var/log/nginx/redirects-access.log vhosts;
    error_log  /var/log/nginx/redirects-error.log;

    rewrite ^ https://$host$request_uri? permanent;
}


server {

        listen 443 ssl;

        server_name jenkins.my-domain;

        ssl_certificate ssl/my-domain.crt;
        ssl_certificate_key ssl/my-domain.key;
        ssl_dhparam ssl/dhparam-2048.pem;
        add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";
        add_header X-Frame-Options SAMEORIGIN;

        access_log  /var/log/nginx/jenkins.my-domain.access.log;
        error_log /var/log/nginx/jenkins.my-domain.error.log;

        location / {
            proxy_set_header        Host $host:$server_port;
            proxy_set_header        X-Real-IP $remote_addr;
            proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header        X-Forwarded-Proto $scheme;
            proxy_redirect http:// https://;
            proxy_pass              http://jenkins;
            # Required for new HTTP-based CLI
            proxy_http_version 1.1;
            proxy_request_buffering off;
            # workaround for https://issues.jenkins-ci.org/browse/JENKINS-45651
            add_header 'X-SSH-Endpoint' 'jenkins.my-domain:22' always;
        }
}


So either I'm blind, or the documentation is somehow wrong?

And indeed, I can see "java.io.IOException: HTTP full-duplex channel timeout" in jenkins log.

This one indeed works:

java -jar jenkins-cli.jar -s http://localhost:8080 -auth user:pass help offline-node

But since I need to execute it from remote, I'd rather connect to https://jenkins.my-domain
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsub...@googlegroups.com.

Devin Nusbaum

unread,
Oct 26, 2017, 5:56:55 PM10/26/17
to jenkins...@googlegroups.com
Some comments on https://issues.jenkins-ci.org/browse/JENKINS-43666 suggest that proxy_buffering off; is needed for SSL connections. (I was also testing nginx reverse proxies in the past and have that setting in my SSL config; might have forgotten to update the SSL docs with my findings) Try setting that option, and if it works I can update the  SSL section of the wiki page.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/476c0ec2-753f-45bd-944b-2f9dcf60deae%40googlegroups.com.

Tomasz Chmielewski

unread,
Oct 26, 2017, 6:06:42 PM10/26/17
to Jenkins Users
That was it! Adding proxy_buffering off helped.

For reference, here is my SSL vhost definition:

server {

        listen 443 ssl;

        server_name jenkins.my-domain;

        ssl_certificate ssl/my-domain.crt;
        ssl_certificate_key ssl/my-domain.key;
        ssl_dhparam ssl/dhparam-2048.pem;
        add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";
        add_header X-Frame-Options SAMEORIGIN;

        access_log  /var/log/nginx/jenkins.my-domain.access.log;
        error_log /var/log/nginx/jenkins.my-domain.error.log;

        location / {
            proxy_set_header        Host $host:$server_port;
            proxy_set_header        X-Real-IP $remote_addr;
            proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header        X-Forwarded-Proto $scheme;
            proxy_redirect http:// https://;
            proxy_pass              http://jenkins;

            # Required for new HTTP-based CLI
            proxy_http_version 1.1;
            proxy_request_buffering off;
            proxy_buffering off;


            # workaround for https://issues.jenkins-ci.org/browse/JENKINS-45651
            # not used for this installation
            #add_header 'X-SSH-Endpoint' 'jenkins.my-domain:22' always;

Devin Nusbaum

unread,
Oct 26, 2017, 6:25:43 PM10/26/17
to jenkins...@googlegroups.com
Great, thanks for confirming! I’ll update the docs.

Thanks,
Devin

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/cb454333-dddf-4186-ac97-66550f856454%40googlegroups.com.

Tomasz Chmielewski

unread,
Oct 27, 2017, 1:03:04 AM10/27/17
to Jenkins Users
Actually, I still didn't find a solution to my problem.

1) I don't understand the purpose of "wait-node-offline"

java -jar jenkins-cli.jar -s https://jenkins.my-domain -auth user:pass wait-node-offline some-node

This will never exit, unless I use "offline-node" first. But "offline-node" exits immediately.

To be precise: actually "wait-node-offline" will crash after some time and will never exit if used through a reverse proxy like nginx. But it will never exit if connecting to jenkins directly.


2) "offline-node" exits immediately and the node is set to "offline":

java -jar jenkins-cli.jar -s https://jenkins.my-domain -auth user:pass offline-node some-node

The builds which were running will continue to run until finished (existing builds are not interrupted) - this is expected.


3) So my reasoning is that I need some kind of:

java -jar jenkins-cli.jar -s https://jenkins.my-domain -auth user:pass list-jobs some-view

But only showing the builds which are currently running (for a given view, or for a given node). Then parse it, and decide if the node is really offline (as in, not accepting any new builds, and not running any builds right now) or not.


Unfortunately, I don't see such an option in https://jenkins.my-domain/cli/

Robert Hales

unread,
Oct 27, 2017, 1:47:37 AM10/27/17
to Jenkins Users
I think wait-offline-node is supposed to just wait until the node is actually offline. So you are supposed to run offline-node, then in your script you can wait-offline-node before you do whatever what you wanted to do after the node was offline. I would assume the node is not configured offline until there are no jobs running on it, but that may be a bad assumption. 

Try kicking off a long-running job (with a sh"sleep 500") and then try to offline the node and then run wait-offline-node to see if it shows it offline immediately, or if it waits until the job finishes. 

If it isn't actually waiting for the jobs to finish, well, that sucks. But the wait-offline-node is still probably supposed to wait for a node to go offline. There may be other things that cause the node to take a long time to go offline. 

Tomasz Chmielewski

unread,
Oct 27, 2017, 7:29:00 AM10/27/17
to Jenkins Users
Unfortunately it doesn't work this way. If there is a long running job, "wait-node-offline" will not wait for it to finish.

# time java -jar jenkins-cli.jar -s https://jenkins.my-domain -auth user:pass offline-node jenkins-node

real    0m1.249s
user    0m0.834s
sys     0m0.049s


At this point, I can see the node is set to offline in Build Executor Status in Jenkins webgui, and the job which was running before starting the command continues to run.


# time java -jar jenkins-cli.jar -s https://jenkins.my-domain -auth user:pass wait-node-offline jenkins-node

real    0m1.949s
user    0m0.817s
sys     0m0.061s


At this point, I can still see the node is set to offline in Build Executor Status in Jenkins webgui, and the job which was running before starting the command continues to run.


So yeah, that sucks, because there doesn't seem to be a good way to offline a jenkins node gracefully.

Robert Hales

unread,
Oct 27, 2017, 11:19:33 AM10/27/17
to Jenkins Users
Looks like you will have to look at writing a groovy script, maybe. You could offline it in a groovy script, then there must be some way to get the status of the node. Can probably put that in a 'while' loop until the node is idle. Then do your other work. You can send a groovy script to the script console through http on the rest API. I can shove you in the right direction, but that's about all the more help I will probably be, though I'd be interested in seeing your script when/if you do it. These links should get you started. 




You could look into the rest API, too. You can offline the node in the api: 


I can't find a way to see the executor status that works. I would think 

http://jenkins/computer/${NODE}/api/json?pretty=true

would do it, but in a quick test, it didn't seem to show different if the executors were busy or not. 

Let me know what you come up with. 
Reply all
Reply to author
Forward
0 new messages