using .each with closure in jenkins pipeline - @NonCPS method

1,788 views
Skip to first unread message

niristotle okram

unread,
Apr 7, 2017, 1:36:29 PM4/7/17
to jenkins...@googlegroups.com
Can anyone say, if the issue of using '.each{ }' , with the closure in jenkins https://issues.jenkins-ci.org/browse/JENKINS-26481 

also affect a NonCPS method?? 

I find it works okay in a master only env. 


niristotle okram

unread,
Apr 9, 2017, 9:29:43 PM4/9/17
to jenkins...@googlegroups.com
Is anyone aware of iterating/looping either using a for / each inside a NonCPS method in a master slave environment? 

I am using the "readFile", DSL to read from the slave's workspace. And i parsing it via the jsonsluper. I am unable to loop through the object. 

Have anyone done this successfully? 

--
Regards
nirish okram

Mark Waite

unread,
Apr 10, 2017, 12:00:46 AM4/10/17
to jenkins...@googlegroups.com
As far as I know, Jesse's advice was to use the more Java like iteration technique, rather than for / each.

Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAPzcO4idZco8oAQcWpRocE5XuRw%3DCrph7oaCqFnQw7-0W-jmAg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

niristotle okram

unread,
Apr 10, 2017, 1:02:39 AM4/10/17
to jenkins...@googlegroups.com
I have seen that mentioned somewhere in the past. But i am baffled as to why things works in the master and fails in the slave/agent. 

Can anyone spot where i am faulting. this have owned me for days now :( ... i am trying to generate an XML file. based on the data provided by a JSON file. 


The pipeline snippet is 

import groovy.xml.*
import groovy.json.JsonSlurper

node('slave1') {
  deleteDir()
  
  stage('Checkout') {
    checkout([$class: 'GitSCM', branches: [[name: '*/master']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [],
    userRemoteConfigs: [[credentialsId: 'b27f7cb2-efa8-496a-90d8-825b9332bf44', url: 'g...@somerepo.git']]])
  }
  
  writeFile file: "MyFile.XML", text: GenerateXML()    //calling the method to generate the XML
  
  println "Generated the manifest XML"
}


@NonCPS
def GenerateXML() {
  /*
  parsing the obj.json file
  */
  def currentws = pwd()
  println currentws
  def jsonSlurper = new JsonSlurper();  
  //def fileReader = new BufferedReader(new FileReader("${currentws}/objects.json"))  //the file location need to change in the actual implementation
  def fileReader = readFile "${currentws}/objects.json"  // ^^avoided the above line to use the pipeline DSL 
  def parsedData = jsonSlurper.parse(fileReader)
  /*
  creating the xml
  */
  // def writer = new FileWriter("${currentws}/sampleManifest123.XML")
  def builder = new StreamingMarkupBuilder()
  builder.encoding = 'UTF-8'
  def xml = builder.bind {
    mkp.xmlDeclaration()
    mkp.declareNamespace('udm.DeploymentPackage' :'http://www.w3.org/2001/XMLSchema')
    mkp.declareNamespace('powercenter.PowercenterXml' :'http://www.w3.org/2001/XMLSchema')
    delegate."udm.DeploymentPackage"(version:'$BUILD_NUMBER', application: "sampleApp"){
      delegate.deployables {
        parsedData.each { index, obj ->
          it."powercenter.PowercenterXml"(name:obj.name, file:obj.file) {
            delegate.scanPlaceholders(true)
            delegate.sourceRepository(obj.sourceRepository)
            delegate.folderNameMap {
              obj.folderNames.each { name, value ->
                it.entry(key:name, value)
              }
            }
            delegate.objectNames {
              delegate.value(obj.objectName)
            }
            delegate.objectTypes {
              delegate.value(obj.objectType)
            }
          }
        }
      }
      delegate.dependencyResolution('LATEST')
      delegate.undeployDependencies(false)
    }
  }
  
}





The input json file looks like:

{"workflows1":
  {
    "name": "/wf_multifolder",
    "file": "release1/wf_multifolder.XML",
    "objectName": "wf_multifolder.XML",
    "objectType": "workflow",
    "sourceRepository": "DEV2",
    "folderNames": { "multifolder": "{{multifolder}}","agent1": "{{agentx}}" }
  },
  "workflows1":
    {
      "name": "/wf_multifolder",
      "file": "release1/wf_multifolder.XML",
      "objectName": "wf_multifolder.XML",
      "objectType": "workflow",
      "sourceRepository": "DEV2",
      "folderNames": { "multifolder": "{{multifolder}}","agent1": "{{agentx}}" }
    }
}






The current behaviour is that, the method just works fine in my groovyConsole but fails big time in jenkins, The file "MyFile.XML"  is created in the workspace. But it have the same content as the input json file... Its not performing the loop. 










On Sun, Apr 9, 2017 at 11:00 PM, Mark Waite <mark.ea...@gmail.com> wrote:
As far as I know, Jesse's advice was to use the more Java like iteration technique, rather than for / each.

Mark Waite

On Sun, Apr 9, 2017 at 7:29 PM niristotle okram <nirish...@gmail.com> wrote:
Is anyone aware of iterating/looping either using a for / each inside a NonCPS method in a master slave environment? 

I am using the "readFile", DSL to read from the slave's workspace. And i parsing it via the jsonsluper. I am unable to loop through the object. 

Have anyone done this successfully? 


On Fri, Apr 7, 2017 at 12:36 PM, niristotle okram <nirish...@gmail.com> wrote:
Can anyone say, if the issue of using '.each{ }' , with the closure in jenkins https://issues.jenkins-ci.org/browse/JENKINS-26481 

also affect a NonCPS method?? 

I find it works okay in a master only env. 





--
Regards
nirish okram

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAPzcO4idZco8oAQcWpRocE5XuRw%3DCrph7oaCqFnQw7-0W-jmAg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtEH%3Dsg2MD6Vxr0WSruZsm%3D9%3D3O%3D6%3DN97LhawpfuY3KPnA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Regards
nirish okram

Stephen Connolly

unread,
Apr 10, 2017, 2:23:18 AM4/10/17
to jenkins...@googlegroups.com
On Mon 10 Apr 2017 at 06:02, niristotle okram <nirish...@gmail.com> wrote:
I have seen that mentioned somewhere in the past. But i am baffled as to why things works in the master and fails in the slave/agent. 

Can anyone spot where i am faulting. this have owned me for days now :( ... i am trying to generate an XML file. based on the data provided by a JSON file. 


The pipeline snippet is 

import groovy.xml.*
import groovy.json.JsonSlurper

node('slave1') {
  deleteDir()
  
  stage('Checkout') {
    checkout([$class: 'GitSCM', branches: [[name: '*/master']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [],
    userRemoteConfigs: [[credentialsId: 'b27f7cb2-efa8-496a-90d8-825b9332bf44', url: 'g...@somerepo.git']]])
  }
  
  writeFile file: "MyFile.XML", text: GenerateXML()    //calling the method to generate the XML
  
  println "Generated the manifest XML"
}


@NonCPS
def GenerateXML() {
  /*
  parsing the obj.json file
  */
  def currentws = pwd()
  println currentws
  def jsonSlurper = new JsonSlurper();  
  //def fileReader = new BufferedReader(new FileReader("${currentws}/objects.json"))  //the file location need to change in the actual implementation

This won't work as groovy always executes on the master. You need to use FilePath to get the remote file

  def fileReader = readFile "${currentws}/objects.json"  // ^^avoided the above line to use the pipeline DSL 

Afaik you can put CPS calls inside nonCPS

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.



--
Regards
nirish okram

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAPzcO4hNr8qFX6PmJjeSE4jhgUwxtxLi6kc8rsirJJa-%2BVArfw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.
--
Sent from my phone

niristotle okram

unread,
Apr 10, 2017, 4:28:13 PM4/10/17
to jenkins...@googlegroups.com
I think i am almost there now...
Have to ripe off certain logic from the noncps method ...and use the readfile and writeFile dsl. 

Keeping fingers cross as i type in :)



For more options, visit https://groups.google.com/d/optout.
--
Sent from mobile device, excuse typos if any.

Baptiste Mathus

unread,
Apr 10, 2017, 4:41:57 PM4/10/17
to jenkins...@googlegroups.com
Resending from this morning, which didn't go through, see below.

2017-04-10 8:23 GMT+02:00 Stephen Connolly <stephen.al...@gmail.com>:

On Mon 10 Apr 2017 at 06:02, niristotle okram <nirish...@gmail.com> wrote:
I have seen that mentioned somewhere in the past. But i am baffled as to why things works in the master and fails in the slave/agent. 

Can anyone spot where i am faulting. this have owned me for days now :( ... i am trying to generate an XML file. based on the data provided by a JSON file. 


The pipeline snippet is 

import groovy.xml.*
import groovy.json.JsonSlurper

node('slave1') {
  deleteDir()
  
  stage('Checkout') {
    checkout([$class: 'GitSCM', branches: [[name: '*/master']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [],
    userRemoteConfigs: [[credentialsId: 'b27f7cb2-efa8-496a-90d8-825b9332bf44', url: 'g...@somerepo.git']]])
  }
  
  writeFile file: "MyFile.XML", text: GenerateXML()    //calling the method to generate the XML
  
  println "Generated the manifest XML"
}


@NonCPS
def GenerateXML() {
  /*
  parsing the obj.json file
  */
  def currentws = pwd()
  println currentws
  def jsonSlurper = new JsonSlurper();  
  //def fileReader = new BufferedReader(new FileReader("${currentws}/objects.json"))  //the file location need to change in the actual implementation

This won't work as groovy always executes on the master. You need to use FilePath to get the remote file

Yes, also, avoid IMO doing fancy XML/JSON processing in Java/Groovy. See Pipeline as an orchestration tool with basic/no programming capabilities and delegate things to CLI tools and so on.
If you wish to do more high level things, then possibly have a look at writing a plugin.

In general, keep your pipeline code as stupid as possible. Also, using external tools will make it easier to test/reuse from your own local dev env.

 

  def fileReader = readFile "${currentws}/objects.json"  // ^^avoided the above line to use the pipeline DSL 

Afaik you can put CPS calls inside nonCPS

No, it's not supported. 

"You may not call regular (CPS-transformed) methods, or Pipeline steps, from a @NonCPS method"

 

[snip]

Gardell, Steven

unread,
Apr 10, 2017, 5:06:43 PM4/10/17
to jenkins...@googlegroups.com

That is certainly where the programming experience leads one. But this is unfortunate. It is rather bizarre to put a complete programming language such as groovy in front of someone and then say “but only use a tiny little bit of it.” Makes one wonder whether using groovy as a framework for this was a mistake from the get-go.


The notion of pushing ‘build’ operations down into scripts is perfectly sensible. The idea of having  to ‘pull’ CI-integration  activities ‘up into a plugin’ seems really unfortunate to me.

 

At any rate my personal experience with the NonCPS mechanism is that one needs to be thoughtful about defining what is inside the function and what is outside (e.g. passed in to the function). So, for example maybe do the file read outside of the function and pass a string into the GenerateXML() function.

 

 

  def fileReader = readFile "${currentws}/objects.json"  // ^^avoided the above line to use the pipeline DSL 

 

Afaik you can put CPS calls inside nonCPS

 

No, it's not supported. 

 

"You may not call regular (CPS-transformed) methods, or Pipeline steps, from a @NonCPS method"

 

 

 

[snip]

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

Reply all
Reply to author
Forward
0 new messages