Pipelines, iterating maps and more headaches

[Norbert Lange]

Hello,

I trying to get comfortable with Pipelines, so far its a rather
unpleasant experience since I cant even get a simple script going.

1) There seem to be some arcane rules on how to iterate over some
builtin Groovy/Java Types within a sandbox. I haven`t found a way that
works without manually allowing the function.

2) Serialization issues for iterators.

3) Language rules seem inconsistent, for example take the createDList
which seems to execute the code differently (throws errrors, need to
define a explicit variable)

4) Are variables from Closures global??? Is this a Groovy thing or a bug???

There seem to be scarce information on how to write useful simple yet
non-trivial pipeline scripts, even the most basic builtin types can`t
be used freely.
I dont know how to use maps, or if there is a prefered way (which
doesnt needs approval). Layering out code into seperate functions to
not step into serialization issues often results in the very same
lines not working, it even appears there are clashes with variable
names across all scopes?. I hope you got some feedback or pointers on
the issues above.

Kind Regards,
Norbert


Example Pipeline Code follows.
This should fetch some additional modules (the url-mapping shouldnt be
coded in the script and come from a configuration file later), a file
.builddepencies should define name and potentially revision for a
project, should be able to override these (thus a map seems
fitting).All repositories should be checked out from the pipeline
script, remaining build can (and will) be run from shell scripts.

-----------------------------------------------------------
// Gonna be a separate, global function when its grown up
def repomap = [
'myscripts': 'ssh://git@localhost:10022/scripts.git',
'myprojekt': 'ssh://git@localhost:10022/myprojekt.git'
]

node {
// Mark the code checkout 'stage'....
stage 'Checkout'

// Get some code from a GitHub repository
//git branch: 'master', url: repomap['myprojekt']
sh('echo "fake checkout: ' + repomap['myprojekt'] + '"')
sh('echo "myscripts=somebranch" > .builddepencies')

// Get name of needed modules
def depmap = [:]
readFile('.builddepencies').eachLine {
def m = (it =~ /([^=]*)=?(.*)/)
depmap[m[0][1].trim()]=m[0][2].trim()
m = null
}

// Just a check so it works (after script approval )
depmap.each {
println 'map: ' + it.key + '=' + it.value
}
// Weird !
println 'it still defined: ' + it.key
// Weirder, no name-scopes in functions?, why is this not working ??
createDList (depmap)

depmap.each {
name = it.key
revision = it.value
it = null // Ok, I could find my way around serialization issues..
dir (name) {
//git url: repomap[name], changelog: false, poll: false
sh('echo "fake checkout: ' + repomap[name] + '"')
}
}
println 'end'
}

@NonCPS
def createDList(depmap) {
// Need to change the iterator name, then this works?
depmap.each {
println 'map: ' + it.key + '=' + it.value
}
}

[Jesse Glick]

On Tuesday, April 26, 2016 at 7:18:55 PM UTC-4, Norbert Lange wrote:

There seem to be some arcane rules on how to iterate over some
builtin Groovy/Java Types within a sandbox. I haven`t found a way that
works without manually allowing the function.

Which methods did you need to approve? We can easily add them to the default whitelist in the Script Security plugin. But anyway

2) Serialization issues for iterators.

`for (x : list) {…}` works as of `workflow-cps` 2.x. Other iterators do not yet work (outside a `@NonCPS` method). Probably fixing them is not hard, just have not gotten to it yet.

 

take the createDList
which seems to execute the code differently (throws errrors, need to
define a explicit variable)

Not sure what this is about. If you find something you think should work which does not work in a minimal reproducible script, please file a bug report for it.

 

Are variables from Closures global?

Local `def` variables in a closure? Not sure what you are referring to here.

The main problem you are presumably hitting is the well-known JENKINS-26481. We are working on a fix, but in the meantime, do not use any method built into Groovy which takes a closure argument, such as `list.each {x -> …}` or `someText.eachLine {line -> …}`. Rather use a Java-style loop. (To be on the safe side, also avoid iterators, meaning use a C- or JDK 1.4-style loop with an index.)

Incidentally `it` does not currently work in closures as noted in JENKINS-33468; use an explicit parameter name instead.

[Norbert Lange]

Am Mittwoch, 27. April 2016 16:41:40 UTC+2 schrieb Jesse Glick:

On Tuesday, April 26, 2016 at 7:18:55 PM UTC-4, Norbert Lange wrote:

There seem to be some arcane rules on how to iterate over some
builtin Groovy/Java Types within a sandbox. I haven`t found a way that
works without manually allowing the function.

Which methods did you need to approve? We can easily add them to the default whitelist in the Script Security plugin. But anyway

The map`s each (at least)

 

2) Serialization issues for iterators.

`for (x : list) {…}` works as of `workflow-cps` 2.x. Other iterators do not yet work (outside a `@NonCPS` method). Probably fixing them is not hard, just have not gotten to it yet

Yes, these issues I can very likely work around. For someone who is new to Groovy and Jenkins sandboxing, a list of preferred methods would be very welcome (the examples from the workflow libs are rather simple). There are atleast 3 different ways to iterate over containers, and several variations of those for maps.

 

 

take the createDList
which seems to execute the code differently (throws errrors, need to
define a explicit variable)

Not sure what this is about. If you find something you think should work which does not work in a minimal reproducible script, please file a bug report for it.

Where? Is that a feature-not-bug of Groovy, an issue in Jenkins or some Plugin? I was hoping for some feedback as I am not proficient in either of those to pinpoint issues.

The code above should be able to explain the issue, the exact same method body in the node scope works fine, the call will result in some message about "it not defined". Similary there seems some issues with name clashes (if variables in functions are named like those in the node scope), but it mightve been some flukes during trial-and-error

 

 

Are variables from Closures global?

Local `def` variables in a closure? Not sure what you are referring to here.

See last point, and the code were I can access the it variable after the closure were its used (Noted with "// Weird !" )

The main problem you are presumably hitting is the well-known JENKINS-26481. We are working on a fix, but in the meantime, do not use any method built into Groovy which takes a closure argument, such as `list.each {x -> …}` or `someText.eachLine {line -> …}`. Rather use a Java-style loop. (To be on the safe side, also avoid iterators, meaning use a C- or JDK 1.4-style loop with an index.)

Could you please post me the preferable code for iterating a map in this way? (Not sure I fully understand the bug )

Incidentally `it` does not currently work in closures as noted in JENKINS-33468; use an explicit parameter name instead.

That seems to be one of the issues I am fighting with, and might be that the supposed name-clashes came from unfocused variations of the code. Strangely it does seem to somewhat work in the node body?

 

Kind Regards,
Norbert

[Brian Ray]

FWIW I recently replaced several C-style loops with for ( x in y ) for iterating over both lists and maps in CPS code and for the most part conversion went fine. There were a couple of CPS sections where I could not use that construct and had to fall back on the C-loops and further do a torturous cast to avoid a serialization error getting keys and values from the map. I want to say Set<Map.Entry<K,V>> caused the exception because AbstractMap.SimpleEntry and .SimpleImmutableEntry are serializable, while Set is not, per the JDK.

for ( int i = 0; i < myMap.size(); i++ ) {
 
  // hacktacular String() cloning to avoid NotSerializableException; also
  // hacktacular Map > Set > Array morph to enable C-style looping
  final foo = new String( myMap.entrySet().toArray()[i].value )

  // do stuff with foo...

}

Nonetheless as mentioned in another part of the script I had no problem using the shorter alternative, nor accessing keys and values using myMap.key and myMap.value. Not sure what the difference is with my more stubborn loop.

[Brian Ray]

Side note: The map keys and values are simply Strings.

[Norbert Lange]

Hi Brian,

every single method in the "final foo...." - including the String Constructor requires approval. I was hoping for a proper subset that would work within the sandbox.

What I ended with (several dozen "Builds" later ) is using a helper function squeezing the map into a list, seems the most sane aproach so far =(

Btw, whats the difference of using "def x" vs "x"?

node {
    ....

    for (it2 in mapToList(depmap)) {

        name = it2[0]

        revision = it2[1]

    }

}

@NonCPS

def mapToList(depmap) {

    def dlist = []

    for (entry in depmap) {

        dlist.add([entry.key, entry.value])

    }

    dlist

}

[Brian Ray]

Nice hackaround. I will try that in the uncooperative parts of my script.

Ah yes, def x vs x. On the face of it the two declarations should be identical--roughly typing x as an Object. But there are different scoping implications for each.

[Patrick Wolf]

Did make use of the "Replay" feature at all to troubleshoot your script, Norbert? Just curious.

https://jenkins.io/blog/2016/04/14/replay-with-pipeline/

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/acfaa0de-8585-44ce-bbd7-ba96cf00022a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Patrick Wolf

Sr. Product Manager

CloudBees

[Craig Rodrigues]

Norm,

Take a look at this my posting from a few months ago:

https://groups.google.com/d/msg/jenkinsci-users/P7VMQQuMdsY/bHfBDSn9GgAJ

That link has pointers to the Groovy Script class and Groovy shell.

I think understanding those two classes will help improve your understanding

of Pipeline scripts, especially with respect to global variables.

I am new to Groovy, and it took me a long time to understand

what is going on with Pipeline scripts, before I started digging into the code to figure things out.

--

Craig

[Norbert Lange]

Nope, I initially used groovysh for rapid testing, until I found out good Groovy in Groovy is not necessarily good Groovy in Jenkins. Then I created a job with a Pipeline Script an ran that violently.

AFAIK, the replay feature isnt mentioned in any of the documents from https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin, you should really mention it there to get people using it (I doubt many people will dig through your blog if they want to start with pipelines)

[Jesse Glick]

On Thu, Apr 28, 2016 at 3:48 AM, Norbert Lange <nola...@gmail.com> wrote:
> the replay feature isnt mentioned in any of the documents from
> https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin, you should
> really mention it there to get people using it

Pipeline documentation in general is in a state of limbo at the moment.

[Jesse Glick]

On Wed, Apr 27, 2016 at 6:00 PM, Norbert Lange <nola...@gmail.com> wrote:
> node {
> ....
> for (it2 in mapToList(depmap)) {
> name = it2[0]
> revision = it2[1]
> }
> }
> @NonCPS
> def mapToList(depmap) {
> def dlist = []
> for (entry in depmap) {
> dlist.add([entry.key, entry.value])
> }
> dlist
> }

This is the proper workaround until I can provide a serialization-safe
version of the JRE’s `Map` iterator.