[JIRA] [openstack-cloud-plugin] (JENKINS-35627) Migrate from 1.x to 2.8 openstack cloud plugin: nodes missed

0 views
Skip to first unread message

lawyard@gmail.com (JIRA)

unread,
Jun 10, 2016, 11:51:01 AM6/10/16
to jenkinsc...@googlegroups.com
LawYard LawYard created an issue
 
Jenkins / Bug JENKINS-35627
Migrate from 1.x to 2.8 openstack cloud plugin: nodes missed
Issue Type: Bug Bug
Assignee: Oliver Gondža
Components: openstack-cloud-plugin
Created: 2016/Jun/10 3:50 PM
Environment: Jenkins 1.651.2
Openstack Cloud plugin 2.8
Priority: Blocker Blocker
Reporter: LawYard LawYard

Hi!

I updated Openstack Cloud plugin from 1.x to 2.8 version. After that all my nodes becomed missed with error:

ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins
java.lang.NullPointerException
	at jenkins.plugins.openstack.compute.JCloudsLauncher.launch(JCloudsLauncher.java:24)
	at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:253)
	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

I suppose that problem can be in nodes configs: 18 strings in prev version vs 153 in new.

Is there a solution wich can help me to return my nodes to work state? I have about 450 nodes and I dont want to lose them(

Thx

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)
Atlassian logo

lawyard@gmail.com (JIRA)

unread,
Jun 10, 2016, 11:52:01 AM6/10/16
to jenkinsc...@googlegroups.com
LawYard LawYard updated an issue
Change By: LawYard LawYard
Hi!

I updated Openstack Cloud plugin from 1.x to 2.8 version. After that all my nodes becomed missed with error:


{code:java}

ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins
java.lang.NullPointerException
at jenkins.plugins.openstack.compute.JCloudsLauncher.launch(JCloudsLauncher.java:24)
at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:253)
at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}


I suppose that problem can be in nodes configs: 18 strings in prev version vs 153 in new. 

Is there a solution wich can help me to return my nodes to work state? I have about 450 nodes and I dont want to lose them(

I have not found information about my issue in changelogs

Thx

ogondza@gmail.com (JIRA)

unread,
Jun 10, 2016, 12:25:03 PM6/10/16
to jenkinsc...@googlegroups.com
Oliver Gondža commented on Bug JENKINS-35627
 
Re: Migrate from 1.x to 2.8 openstack cloud plugin: nodes missed

Are those all openstack slaves? Are they expected to survive restart? If so, I presume the problem is the class have a new field that is now initialized with null. This is a bug.

lawyard@gmail.com (JIRA)

unread,
Jun 10, 2016, 2:45:02 PM6/10/16
to jenkinsc...@googlegroups.com

Oliver, yes - all of my slaves are openstack slaves and they expected to survive restart. Do u have some ideas how to help me to migrate before (if) the new version (with fix) will be released? Maybe some nodes config.xml changes?

ogondza@gmail.com (JIRA)

unread,
Jun 10, 2016, 4:21:01 PM6/10/16
to jenkinsc...@googlegroups.com

That would be non-trivial. Can you provide an xml excerpt from confg.xml with slave configuration I can use to reproduce the problem? I guess your best option is to downgrade to pre-2.1 version until this is fixed.

lawyard@gmail.com (JIRA)

unread,
Jun 10, 2016, 4:25:01 PM6/10/16
to jenkinsc...@googlegroups.com

Slave config:

<?xml version='1.0' encoding='UTF-8'?>
<jenkins.plugins.openstack.compute.JCloudsSlave plugin="openstack-cloud@2.8">
  <name>vanilla-trusty-large-sas-a08</name>
  <remoteFS>/jenkins</remoteFS>
  <numExecutors>1</numExecutors>
  <mode>EXCLUSIVE</mode>
  <retentionStrategy class="hudson.slaves.RetentionStrategy$Always"/>
  <launcher class="jenkins.plugins.openstack.compute.JCloudsLauncher"/>
  <label>vanilla-trusty-large-sas vanilla-trusty-large</label>
  <nodeProperties/>
  <userId>anonymous</userId>
  <cloudName>OpenstackCloud_SAS</cloudName>
  <options>
    <jvmOptions>-Dfile.encoding=UTF-8 -Xmx2G</jvmOptions>
    <credentialsId>068643d6-9c43-4d1b-ae00-e2ecdcc5cbc6</credentialsId>
    <slaveType>SSH</slaveType>
    <retentionTime>0</retentionTime>
  </options>
</jenkins.plugins.openstack.compute.JCloudsSlave>

lawyard@gmail.com (JIRA)

unread,
Jun 10, 2016, 4:31:01 PM6/10/16
to jenkinsc...@googlegroups.com

About downgrading to pre-2.1: does it support Openstack Liberty and Mitaka?

lawyard@gmail.com (JIRA)

unread,
Jun 14, 2016, 3:28:01 AM6/14/16
to jenkinsc...@googlegroups.com

I have Installed 2.0 version - has the same error

This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

lawyard@gmail.com (JIRA)

unread,
Jun 23, 2016, 9:33:04 AM6/23/16
to jenkinsc...@googlegroups.com

I wrote script for migration - it must be executed after updating plugin throw script console. It scan all nodes and creates new xml configs within folder /var/lib/jenkins/nodes_ported. Then you can backup origin `nodes` folder and rename `nodes_ported` to `nodes` and restart jenkins. Its worked for me. This script has some customization but thats better than nothing)

import groovy.xml.MarkupBuilder

nodes_xmls = [:]

// ------------------------------------------------------
// ============== GENERATING NODES XMLS =================
// ------------------------------------------------------

// get all cloud nodes from openstack
cloud_nodes = [:]
for (cloud in Jenkins.instance.clouds.findAll{!it.name.contains('Compound')}) {
    cloud_nodes.put(cloud.name, cloud.openstack.client.compute().servers().list(true))
}

for (dead_slave in hudson.model.Hudson.instance.slaves.findAll{!it.name.contains('Compound')}) {
    // Get dead_slave available properties
    slave_executors = dead_slave.getNumExecutors()
    slave_retention_policy = dead_slave.getRetentionStrategy().toString().split('@')[0]
    slave_cloud_name = dead_slave.cloudName
    slave_labels = dead_slave.getLabelString()
    slave_cred_id = dead_slave.options.credentialsId
    slave_jvm_opts = dead_slave.options.jvmOptions
    slave_remote_fs = dead_slave.getRemoteFS()
    slave_mode = dead_slave.getMode()

    cloud = Jenkins.instance.clouds.find{it.name == dead_slave.cloudName}
    cloud_prefix = cloud.name.split('_')[1]
    node = cloud_nodes[slave_cloud_name].find{it.name == dead_slave.name}

    def writer = new StringWriter()
    def xml = new MarkupBuilder(writer)

    // ==== get IP addrs:
    slave_addrs = [:]
    node.getAddresses().getAddresses().each { k, v ->
        slave_addrs.put('ipv4', v.find { it.getVersion() == 4 }.getAddr())
        slave_addrs.put('ipv6', v.find { it.getVersion() == 6 }.getAddr())
    }
    node_template_name = slave_labels.split().find { it.toUpperCase().contains(cloud_prefix) }
    node_template = cloud.getTemplate(node_template_name).getRawSlaveOptions()

    // ======================== Header ================================
    xml.'jenkins.plugins.openstack.compute.JCloudsSlave'(plugin: 'openstack-cloud@2.8') {
        name(node.name)
        remoteFS(slave_remote_fs)
        numExecutors(slave_executors)
        mode(slave_mode)
        retentionStrategy(class: slave_retention_policy)
        // ======================== Launcher ================================
        launcher(class: 'jenkins.plugins.openstack.compute.JCloudsLauncher') {
            launcher(class: 'hudson.plugins.sshslaves.SSHLauncher', plugin: 'ssh-s...@1.10') {
                if (slave_addrs['ipv4']) {
                    host(slave_addrs['ipv4'])
                }
                else {
                    host(slave_addrs['ipv6'])
                }
                port(22)
                credentialsId(slave_cred_id)
                if (slave_jvm_opts) {
                    jvmOptions(slave_jvm_opts)
                }
                launchTimeoutSeconds(1200)
                maxNumRetries(5)
                retryWaitTime(15)
            }
        }
        // ======================== Label ================================
        label(slave_labels)
        // ======================== nodeProperties ================================
        nodeProperties() {
            'hudson.slaves.EnvironmentVariablesNodeProperty'() {
                envVars(serialization: 'custom') {
                    'unserializable-parents'()
                    'tree-map'() {
                        'default'() {
                            comparator(class: 'hudson.util.CaseInsensitiveComparator')
                        }
                        'int'(1)
                        string('OPENSTACK_PUBLIC_IP')
                        if (slave_addrs['ipv4']) {
                            string(slave_addrs['ipv4'])
                        }
                        else {
                            string(slave_addrs['ipv6'])
                        }
                    }
                }
            }
        }
        // ======================== userId ================================
        userId(node.userId)
        // ======================== MetaData NovaServer ================================
        metadata(class: 'org.openstack4j.openstack.compute.domain.NovaServer') {
            id(node.id)
            name(node.name)
            addresses() {
                addresses() {
                    node.getAddresses().getAddresses().each { key, value ->
                        entry() {
                            string(key)
                            list() {
                                for (val in value) {
                                    'org.openstack4j.openstack.compute.domain.NovaAddresses_-NovaAddress'() {
                                        macAddr(val.getMacAddr())
                                        version(val.getVersion())
                                        addr(val.getAddr())
                                        type(val.getType())
                                    }
                                }
                            }
                        }
                    }
                }
            }

            links() {
                for (link in node.links) {
                    'org.openstack4j.openstack.common.GenericLink'() {
                        rel(link.getRel())
                        href(link.getHref())
                    }
                }
            }
            image(class: 'linked-hash-map') {
                entry() {
                    string('id')
                    string(node.image.id)
                }
                entry() {
                    string('links')
                    list() {
                        'linked-hash-map'() {
                            entry() {
                                string('href')
                                string(node.image.links.find { it.getRel() == 'bookmark' }.getHref())
                            }
                            entry() {
                                string('rel')
                                string('bookmark')
                            }
                        }
                    }
                }
            }
            flavor() {
                id(node.getFlavor().getId())
                ephemeral(node.getFlavor().getEphemeral())
                swap(node.getFlavor().getSwap())
                rxtxFactor(node.getFlavor().getRxtxFactor())
                isPublic(node.getFlavor().isPublic())
                links() {
                    'org.openstack4j.openstack.common.GenericLink'() {
                        rel(node.links.find { it.getRel() == 'bookmark' }.getRel())
                        href(node.links.find { it.getRel() == 'bookmark' }.getHref())
                    }
                }
            }
            accessIPv4(node.accessIPv4)
            accessIPv6(node.accessIPv6)
            configDrive(node.configDrive)
            status(node.status)
            progress(node.progress)
            tenantId(node.tenantId)
            userId(node.userId)
            keyName(node.keyName)
            hostId(node.hostId)
            updated(node.updated)
            created(node.created)
            metadata(class: 'linked-hash-map') {
                node.metadata.each { k, v ->
                    entry() {
                        string(k)
                        string(v)
                    }
                }
            }
            powerstate(node.getPowerState())
            vmState(node.getVmState())
            diskConfig(node.getDiskConfig())
            availabilityZone(node.getAvailabilityZone())
            launchedAt(node.getLaunchedAt())
            osExtendedVolumesAttached()
        }
        // ======================== CloudName ================================
        cloudName(cloud.name)
        // ======================== options ================================
        options() {
            imageId(node_template.imageId)
            hardwareId(node_template.hardwareId)
            networkId(node_template.networkId)
            userDataId()
            instanceCap(cloud.slaveOptions.instanceCap)
            securityGroups(node_template.securityGroups)
            startTimeout(cloud.slaveOptions.startTimeout)
            keyPairName(node_template.keyPairName)
            numExecutors(slave_executors)
            if (slave_jvm_opts) {
                jvmOptions(slave_jvm_opts)
            }
            fsRoot(slave_remote_fs)
            credentialsId(slave_cred_id)
            slaveType('SSH')
            retentionTime(30)
        }
        // ======================== provisioningId ================================
        provisioningId(plugin: 'cloud-stats@0.1') {
            cloudName(cloud.name)
            templateName(node_template_name)
            fingerprint(System.identityHashCode(node.name) ^ (int) System.currentTimeMillis())
        }
    }
    // ======== Put Node XML config to map `nodes_xml`
    nodes_xmls.put(dead_slave.name, writer)
}

// ----------------------------------------------------------
// ============== WRITING CONFIGS TO FILESYSTEM =================
// ----------------------------------------------------------

nodes_folder = '/var/lib/jenkins/nodes_ported/'

// ----- writing nodes ----

nodes_xmls.each{nodename, xml_config ->
    node_folder = nodes_folder + nodename
    config_full_path = node_folder + '/config.xml'
    // create folder for config
    new File(node_folder).mkdirs()
    // write config to folder
    conf_file = new File(config_full_path)
    conf_file.write("<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n")
    conf_file.append(xml_config)
}

ogondza@gmail.com (JIRA)

unread,
Jun 30, 2016, 9:32:01 AM6/30/16
to jenkinsc...@googlegroups.com

I am really sorry you had to put that much effort into migration. Though, it is sort of worrying that such a large deployment depend on features like this to operate. I wonder, why you can not respin those machines?

I have committed an improvement[1] for the migration from ancient releases just today.

[1] https://github.com/jenkinsci/openstack-cloud-plugin/commit/02096a61a85262ebfea41fec5674cc8de6a774da

ogondza@gmail.com (JIRA)

unread,
Jun 30, 2016, 9:34:01 AM6/30/16
to jenkinsc...@googlegroups.com
Oliver Gondža edited a comment on Bug JENKINS-35627
I am really sorry you had to put that much effort into migration. Though, it is sort of worrying that such a large deployment depend on features like this to operate. I wonder, why you can not respin those machines?

I have committed an improvement[1] for the migration from ancient releases just today.

EDIT: If you still have a backup of your original config, it would be great if you can try the migration one more time to see if there are any further problems. Note that the config.xml you have provided earlier was already crunched by 2.8 and therefore unusable for testing purposes so I generated my own.

[1] https://github.com/jenkinsci/openstack-cloud-plugin/commit/02096a61a85262ebfea41fec5674cc8de6a774da

ogondza@gmail.com (JIRA)

unread,
Jul 11, 2016, 6:31:01 AM7/11/16
to jenkinsc...@googlegroups.com
Oliver Gondža resolved as Fixed
 

Improvement released in 2.9

Change By: Oliver Gondža
Status: Open Resolved
Resolution: Fixed

lawyard@gmail.com (JIRA)

unread,
Jul 11, 2016, 7:02:03 AM7/11/16
to jenkinsc...@googlegroups.com
LawYard LawYard commented on Bug JENKINS-35627
 
Re: Migrate from 1.x to 2.8 openstack cloud plugin: nodes missed

Hello, Oliver!

Sorry for the long answer. I have not backup of my original config, so cant try the migration(

Can we mark this issue as solved?

ogondza@gmail.com (JIRA)

unread,
Jul 11, 2016, 7:06:01 AM7/11/16
to jenkinsc...@googlegroups.com

lawyard@gmail.com (JIRA)

unread,
Jul 11, 2016, 7:10:01 AM7/11/16
to jenkinsc...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages