[JIRA] [workflow-plugin] (JENKINS-27421) java.util.ArrayList$Itr is not Serializable

281 views
Skip to first unread message

achittur@mercenarytechnology.com (JIRA)

unread,
May 6, 2015, 10:57:04 PM5/6/15
to jenkinsc...@googlegroups.com
A C commented on Bug JENKINS-27421
 
Re: java.util.ArrayList$Itr is not Serializable

bump The inability to use collection iterators at all is a very significant limitation.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)
Atlassian logo

jglick@cloudbees.com (JIRA)

unread,
Jun 16, 2015, 2:18:01 PM6/16/15
to jenkinsc...@googlegroups.com

Workarounds:

· Use a plain C-style for-loop.
· Wrap the code in a method marked @NonCps, assuming it does not invoke anything interruptible (incl. calling Workflow steps).

owen@nerdnetworks.org (JIRA)

unread,
Jul 14, 2015, 6:54:02 PM7/14/15
to jenkinsc...@googlegroups.com

Being able to write:

node('linux_slave', 'windows_slave')

{ // some steps to run in parallel on both slaves }

would be fantastic.

jglick@cloudbees.com (JIRA)

unread,
Aug 12, 2015, 7:49:01 AM8/12/15
to jenkinsc...@googlegroups.com

owenmehegan you can already do that using

def labels = ['linux', 'windows']
for (int i = 0; i < labels.size(); i++) {
  node(labels.get(i)) {
    // something, though you presumably need to select sh vs. bat here
  }
}

jglick@cloudbees.com (JIRA)

unread,
Sep 10, 2015, 9:17:04 AM9/10/15
to jenkinsc...@googlegroups.com
Jesse Glick started work on Bug JENKINS-27421
 
Change By: Jesse Glick
Status: Open In Progress

scm_issue_link@java.net (JIRA)

unread,
Sep 11, 2015, 4:44:04 PM9/11/15
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Jesse Glick
Path:
aggregator/src/test/java/org/jenkinsci/plugins/workflow/PersistenceFailureTest.java
aggregator/src/test/java/org/jenkinsci/plugins/workflow/PersistenceProblemStep.java
aggregator/src/test/java/org/jenkinsci/plugins/workflow/PersistenceProblemStepExecution.java
aggregator/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
aggregator/src/test/java/org/jenkinsci/plugins/workflow/WorkflowTest.java
http://jenkins-ci.org/commit/workflow-plugin/61b7d70b1238d9d66dea22f497cac26debaaf88e
Log:
Merge pull request #206 from jglick/Itr-JENKINS-27421

JENKINS-27421 Investigating ArrayList.Itr problem

Compare: https://github.com/jenkinsci/workflow-plugin/compare/a24f13091cf8...61b7d70b1238

scm_issue_link@java.net (JIRA)

unread,
Sep 11, 2015, 4:44:04 PM9/11/15
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Jesse Glick
Path:

aggregator/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
http://jenkins-ci.org/commit/workflow-plugin/e916b5e9f35b00db679360dc29e42658d566efb7
Log:
JENKINS-27421 Reproduced in test.

steve.sides@oracle.com (JIRA)

unread,
Oct 12, 2015, 2:03:04 PM10/12/15
to jenkinsc...@googlegroups.com

Here's another context. I was reading some mercurial respotories with source and subdir (to check out to) from a json file:
for(repomap in jsonrepos.configs.repos) {
for ( key in repomap.keySet() ){
source = repomap.get(key)['source']
subdir = repomap.get(key)['subdir']
checkout([$class: 'MercurialSCM', credentialsId: '', installation: '(Default)', source: "$

{source}", subdir: "${subdir}"])
}

This looked nice and elegant, but I hit this exception, so I removed the foreach loops (I include the json steps here):

def slurper = new JsonSlurper()
def jsonText = readFile 'flow-configs/prebuild.json'
jsonrepos = slurper.parseText( jsonText )
repoCount = jsonrepos.configs.repos.size
println "repositories: ${repoCount}"
for ( i = 0 ; i < repoCount ; i++ ) {
rmap = jsonrepos.configs.repos[i]
key = rmap.keySet().toList().getAt(0)
source = rmap.get(key)['source']
subdir = rmap.get(key)['subdir']
println(" source: ${source}

subdir: $

{subdir}")
checkout([$class: 'MercurialSCM', credentialsId: '', installation: '(Default)', source: "${source}", subdir: "${subdir}

"])
}
It's fine up to the println statement. If you comment out the checkout line, it's all good and prints all the repsitories and subdir info, but if I uncomment out the checkout line I get this exception, which is similar to the rest reported except for the slurper part (which is curious):

java.io.NotSerializableException: groovy.json.JsonSlurper
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:860)
at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50)
at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
at java.util.HashMap.internalWriteEntries(HashMap.java:1777)
at java.util.HashMap.writeObject(HashMap.java:1354)
at sun.reflect.GeneratedMethodAccessor199.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:271)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:976)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50)
at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
at java.util.TreeMap.writeObject(TreeMap.java:2434)
at sun.reflect.GeneratedMethodAccessor206.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:271)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:976)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)
at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)
at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverWriter.writeObject(RiverWriter.java:132)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:344)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:328)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:303)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$000(CpsThreadGroup.java:71)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:180)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:178)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:47)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: an exception which occurred:
in field locals
in field parent
in field parent
in field caller
in field e
in field program
in field threads
in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@509e61d4
Finished: FAILURE

steve.sides@oracle.com (JIRA)

unread,
Oct 12, 2015, 2:05:06 PM10/12/15
to jenkinsc...@googlegroups.com
steve sides edited a comment on Bug JENKINS-27421
Here's another context. I was reading some mercurial respotories with source and subdir (to check out to) from a json file:
<pre>
</pre>

steve.sides@oracle.com (JIRA)

unread,
Oct 12, 2015, 2:06:05 PM10/12/15
to jenkinsc...@googlegroups.com

steve.sides@oracle.com (JIRA)

unread,
Oct 12, 2015, 2:07:04 PM10/12/15
to jenkinsc...@googlegroups.com
steve sides edited a comment on Bug JENKINS-27421
Here's another context. I was reading some mercurial respotories with source and subdir (to check out to) from a json file  (sorry for the formatting, not sure how to get jira to keep my spacing) :

{ { quote} for(repomap in jsonrepos.configs.repos) {

    for ( key in repomap.keySet() ){
        source = repomap.get(key)['source']
        subdir = repomap.get(key)['subdir']
        checkout([$class: 'MercurialSCM', credentialsId: '', installation: '(Default)', source: "${source}", subdir: "${subdir}"])
   } {quote}
}}

steve.sides@oracle.com (JIRA)

unread,
Oct 12, 2015, 2:08:03 PM10/12/15
to jenkinsc...@googlegroups.com

steve.sides@oracle.com (JIRA)

unread,
Oct 12, 2015, 2:13:02 PM10/12/15
to jenkinsc...@googlegroups.com
steve sides edited a comment on Bug JENKINS-27421
Here's another context. I was reading some mercurial respotories with source and subdir (to check out to) from a json file (sorry for the formatting, not sure how to get jira to keep my spacing):


{code:java}
for(repomap in jsonrepos.configs.repos) {
    for ( key in repomap.keySet() ){
        source = repomap.get(key)['source']
        subdir = repomap.get(key)['subdir']
        checkout([$class: 'MercurialSCM', credentialsId: '', installation: '(Default)', source: "${source}", subdir: "${subdir}"])
   }
{code}


This looked nice and elegant, but I hit this exception, so I removed the foreach loops (I include the json steps here):


{code:java}
def slurper = new JsonSlurper()
def jsonText = readFile 'flow-configs/prebuild.json'
jsonrepos = slurper.parseText( jsonText )
repoCount = jsonrepos.configs.repos.size
println "repositories: ${repoCount}"
for ( i = 0 ; i < repoCount ; i++ ) {
    rmap = jsonrepos.configs.repos[i]
    key = rmap.keySet().toList().getAt(0)
    source = rmap.get(key)['source']
    subdir = rmap.get(key)['subdir']
    println("  source: ${source}  subdir: ${subdir}")
    checkout([$class: 'MercurialSCM', credentialsId: '', installation: '(Default)', source: "${source}", subdir: "${subdir}"])
}
{code}

steve.sides@oracle.com (JIRA)

unread,
Oct 12, 2015, 2:14:04 PM10/12/15
to jenkinsc...@googlegroups.com

steve.sides@oracle.com (JIRA)

unread,
Oct 12, 2015, 2:14:05 PM10/12/15
to jenkinsc...@googlegroups.com
steve sides edited a comment on Bug JENKINS-27421
Here's another context. I was reading some mercurial  respotories  repositories  with source and subdir (to check out to) from a json file:

steve.sides@oracle.com (JIRA)

unread,
Oct 12, 2015, 2:20:06 PM10/12/15
to jenkinsc...@googlegroups.com
steve sides edited a comment on Bug JENKINS-27421
Here's another context. I was reading some mercurial repositories with source and subdir (to check out to) from a json file:


{code:java}
for(repomap in jsonrepos.configs.repos) {
    for ( key in repomap.keySet() ){
        source = repomap.get(key)['source']
        subdir = repomap.get(key)['subdir']
        checkout([$class: 'MercurialSCM', credentialsId: '', installation: '(Default)', source: "${source}", subdir: "${subdir}"])
   }
{code}


This looked nice and elegant, but I hit this exception  noted in this bug , so I removed the foreach loops (I include the json steps here):



{code:java}
def slurper = new JsonSlurper()
def jsonText = readFile 'flow-configs/prebuild.json'
jsonrepos = slurper.parseText( jsonText )
repoCount = jsonrepos.configs.repos.size
println "repositories: ${repoCount}"
for ( i = 0 ; i < repoCount ; i++ ) {
    rmap = jsonrepos.configs.repos[i]
    key = rmap.keySet().toList().getAt(0)
    source = rmap.get(key)['source']
    subdir = rmap.get(key)['subdir']
    println("  source: ${source}  subdir: ${subdir}")
    checkout([$class: 'MercurialSCM', credentialsId: '', installation: '(Default)', source: "${source}", subdir: "${subdir}"])
}
{code}

It's fine up to the println statement. If you comment out the checkout line, it's all good and prints all the repsitories and subdir info, but if I uncomment out the checkout line I get this exception, which is similar to the rest reported except for the slurper part (which is curious) : .
This may be a different bug(???), but the stack trace seems to be the same (except the json.JsonSlurper) as other bugs referred to as another instance of this bug.

steve.sides@oracle.com (JIRA)

unread,
Oct 13, 2015, 1:11:04 PM10/13/15
to jenkinsc...@googlegroups.com
steve sides edited a comment on Bug JENKINS-27421
Here's another context  where moving away from foreach was not effective .  
 I was reading some mercurial repositories with source and subdir (to check out to) from a json file:

{code:java}
for(repomap in jsonrepos.configs.repos) {
    for ( key in repomap.keySet() ){
        source = repomap.get(key)['source']
        subdir = repomap.get(key)['subdir']
        checkout([$class: 'MercurialSCM', credentialsId: '', installation: '(Default)', source: "${source}", subdir: "${subdir}"])
   }
{code}


This looked nice and elegant, but I hit this exception noted in this bug, so I removed the foreach loops (I include the json steps here):



{code:java}
def slurper = new JsonSlurper()
def jsonText = readFile 'flow-configs/prebuild.json'
jsonrepos = slurper.parseText( jsonText )
repoCount = jsonrepos.configs.repos.size
println "repositories: ${repoCount}"
for ( i = 0 ; i < repoCount ; i++ ) {
    rmap = jsonrepos.configs.repos[i]
    key = rmap.keySet().toList().getAt(0)
    source = rmap.get(key)['source']
    subdir = rmap.get(key)['subdir']
    println("  source: ${source}  subdir: ${subdir}")
    checkout([$class: 'MercurialSCM', credentialsId: '', installation: '(Default)', source: "${source}", subdir: "${subdir}"])
}
{code}

It's fine up to the println statement. If you comment out the checkout line, it's all good and prints all the repsitories and subdir info, but if I uncomment out the checkout line I get this exception, which is similar to the rest reported except for the slurper part (which is curious).

jglick@cloudbees.com (JIRA)

unread,
Oct 14, 2015, 11:09:27 AM10/14/15
to jenkinsc...@googlegroups.com

steve sides unrelated user error. You cannot use an unserializable class such as JsonSlurper from a flow script unless it is contained entirely within a method marked @NonCPS. See the tutorial.

svanoort@cloudbees.com (JIRA)

unread,
Nov 13, 2015, 12:27:01 PM11/13/15
to jenkinsc...@googlegroups.com

Jesse Glick I've got a workaround for how to handle map iteration with Workflow DSL here. A bit ugly but it works:

import com.cloudbees.groovy.cps.NonCPS

@NonCPS
List<Map.Entry> get_map_entries(map) {
    // This is harder than it seems, toArray and naive list construction don't work as expected
    // Also set iterators are forbidden, so you need an indexed collection
    def myarray = []
    myarray.addAll(map.entrySet())
    return myarray
}

node {
  def test_envs = [:]
   test_envs["debian:wheezy"] = ["echo goober"]
   def entries = get_map_entries(test_envs)
   
   for (int i=0; i<entries.size(); i++){
       String key = entries.get(i).key
       String value =  entries.get(i).value
       echo "Key $key and value $value"
   }
}

svanoort@cloudbees.com (JIRA)

unread,
Nov 13, 2015, 12:31:04 PM11/13/15
to jenkinsc...@googlegroups.com

Worth noting (may be logged elsewhere or user error), but trying to do toArray on the entrySet to get an array output easily like so:

def myarray = map.entrySet().toArray()

Will give this error:
hudson.remoting.ProxyException: groovy.lang.MissingMethodException: No signature of method: [Ljava.lang.Object;.get() is applicable for argument types: (java.lang.Integer) values: [0]
Possible solutions: getAt(java.lang.Integer), grep(), getAt(java.lang.String), grep(java.lang.Object), getAt(java.util.Collection), getAt(groovy.lang.ObjectRange)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:55)
at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:46)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:15)
at WorkflowScript.run(WorkflowScript:31)
at Unknown.Unknown(Unknown)
at __cps.transform__(Native Method)
at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:69)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:106)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:79)
at sun.reflect.GeneratedMethodAccessor745.invoke(Unknown Source)


at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)

at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
at com.cloudbees.groovy.cps.impl.LocalVariableBlock$LocalVariable.get(LocalVariableBlock.java:33)
at com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30)
at com.cloudbees.groovy.cps.impl.LocalVariableBlock.evalLValue(LocalVariableBlock.java:22)
at com.cloudbees.groovy.cps.LValueBlock$BlockImpl.eval(LValueBlock.java:55)
at com.cloudbees.groovy.cps.LValueBlock.eval(LValueBlock.java:16)
at com.cloudbees.groovy.cps.Next.step(Next.java:58)
at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:145)
at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:164)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:274)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$000(CpsThreadGroup.java:74)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:183)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:181)


at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:47)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Finished: FAILURE

svanoort@cloudbees.com (JIRA)

unread,
Nov 13, 2015, 12:32:03 PM11/13/15
to jenkinsc...@googlegroups.com
Sam Van Oort edited a comment on Bug JENKINS-27421
[~jglick]  I've got a workaround for how to handle map iteration with Workflow DSL here.  A bit ugly but it works:


{code}

import com.cloudbees.groovy.cps.NonCPS

@NonCPS
List<Map.Entry> get_map_entries(map) {
    // This is harder than it seems, toArray  and naive list construction don  doesn 't work as expected  and there are other gotchas to know
    // Also set iterators are forbidden, so you need an indexed collection
    def myarray = []
    myarray.addAll(map.entrySet())
    return myarray
}

node {
  def test_envs = [:]
   test_envs["debian:wheezy"] = ["echo goober"]
   def entries = get_map_entries(test_envs)
   
   for (int i=0; i<entries.size(); i++){
       String key = entries.get(i).key
       String value =  entries.get(i).value
       echo "Key $key and value $value"
   }
}
{code}

jglick@cloudbees.com (JIRA)

unread,
Nov 13, 2015, 12:58:03 PM11/13/15
to jenkinsc...@googlegroups.com

The MissingMethodException is probably fixable by using entries[i] rather than entries.get(i).

jglick@cloudbees.com (JIRA)

unread,
Nov 13, 2015, 1:03:01 PM11/13/15
to jenkinsc...@googlegroups.com

NonCPS need not be imported—Workflow scripts get it by default.

jglick@cloudbees.com (JIRA)

unread,
Nov 13, 2015, 1:10:02 PM11/13/15
to jenkinsc...@googlegroups.com

If you use an asynchronous step

sh "echo Key $key and value $value"

you will see that your workaround does not really work:

Started by user anonymous
[Workflow] Allocate node : Start
Running on master in …
[Workflow] node {
[Workflow] sh
[flow] Running shell script
[Workflow] } //node
[Workflow] Allocate node : End
[Workflow] End of Workflow
java.io.NotSerializableException: java.util.LinkedHashMap$Entry
	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:860)
	at …
	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
	at java.util.HashMap.internalWriteEntries(HashMap.java:1777)
	at java.util.HashMap.writeObject(HashMap.java:1354)
	at …
	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
	at java.util.TreeMap.writeObject(TreeMap.java:2434)
	at …
	at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverWriter.writeObject(RiverWriter.java:132)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:347)
	at …
Caused by: an exception which occurred:
	in field locals
	in field parent
	in field parent
	in field caller
	in field e
	in field program
	in field threads
	in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@3e45e287
Finished: FAILURE

Here is a working script:

@NonCPS
List<List<Object>> get_map_entries(map) {
    map.collect {k, v -> [k, v]}
}
node {
   def mymap = [a: 1, b: 2]
   def entries = get_map_entries(mymap)
   for (int i=0; i<entries.size(); i++){
       String key = entries[i][0]
       String value =  entries[i][1]
       sh "echo Key $key and value $value"
   }
}

svanoort@cloudbees.com (JIRA)

unread,
Nov 13, 2015, 3:48:01 PM11/13/15
to jenkinsc...@googlegroups.com

Notable limitation: the closure-based solution can trigger script-security violation if you're running workflow scripts from SCM:

org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods collect java.util.Map groovy.lang.Closure
at

Since the collect method takes a closure, AFAICT (and might be wrong) it looks like that could be unsafe to whitelist.

Probably completely doable to find a non-closure approach to this though, just don't have one in hand right now.

svanoort@cloudbees.com (JIRA)

unread,
Nov 13, 2015, 3:50:02 PM11/13/15
to jenkinsc...@googlegroups.com
Sam Van Oort edited a comment on Bug JENKINS-27421
Notable limitation:  the closure-based solution can trigger script-security violation if you're running workflow scripts from SCM:

org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods collect java.util.Map groovy.lang.Closure
at

Since the collect method takes a closure, AFAICT (and might be wrong) it looks like that *could* be unsafe to whitelist.

 Probably
Also  probably  completely doable to find a non-closure approach to this though, just don't have one in hand right now. 

jglick@cloudbees.com (JIRA)

unread,
Nov 13, 2015, 4:18:02 PM11/13/15
to jenkinsc...@googlegroups.com

That method is fine to whitelist.

svanoort@cloudbees.com (JIRA)

unread,
Nov 13, 2015, 4:20:01 PM11/13/15
to jenkinsc...@googlegroups.com

Yes, with note that it is safe because the passed closure is subject to the security rules.

andrew.bayer@gmail.com (JIRA)

unread,
Dec 11, 2015, 1:20:01 PM12/11/15
to jenkinsc...@googlegroups.com

Is this related to JENKINS-26481 or is it a separate cause?

jglick@cloudbees.com (JIRA)

unread,
Jan 11, 2016, 6:27:06 PM1/11/16
to jenkinsc...@googlegroups.com

jglick@cloudbees.com (JIRA)

unread,
Apr 2, 2016, 10:02:02 PM4/2/16
to jenkinsc...@googlegroups.com

Blocking at least the initially attempted use case from JENKINS-26481.

scm_issue_link@java.net (JIRA)

unread,
Apr 2, 2016, 11:31:09 PM4/2/16
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Jesse Glick
Path:
cps/src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java
cps/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
http://jenkins-ci.org/commit/workflow-plugin/a990468a828e64961eda24332673c9cb8835c629
Log:
[FIXED JENKINS-27421] Producing a safe serialization replacement for ArrayList$Itr,

scm_issue_link@java.net (JIRA)

unread,
Apr 2, 2016, 11:31:14 PM4/2/16
to jenkinsc...@googlegroups.com
SCM/JIRA link daemon resolved as Fixed
 
Jenkins / Bug JENKINS-27421
java.util.ArrayList$Itr is not Serializable
Change By: SCM/JIRA link daemon
Status: In Progress Resolved
Resolution: Fixed

jglick@cloudbees.com (JIRA)

unread,
Apr 2, 2016, 11:31:15 PM4/2/16
to jenkinsc...@googlegroups.com
Jesse Glick reopened an issue
 

Not in master yet, be patient JIRA link daemon…

Change By: Jesse Glick
Resolution: Fixed
Status: Resolved Reopened

jglick@cloudbees.com (JIRA)

unread,
Apr 2, 2016, 11:31:18 PM4/2/16
to jenkinsc...@googlegroups.com
Jesse Glick started work on Bug JENKINS-27421
 
Change By: Jesse Glick
Status: Open In Progress

jglick@cloudbees.com (JIRA)

unread,
Apr 2, 2016, 11:31:18 PM4/2/16
to jenkinsc...@googlegroups.com

scm_issue_link@java.net (JIRA)

unread,
Apr 4, 2016, 3:01:09 PM4/4/16
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Jesse Glick
Path:

cps/pom.xml
cps/src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java
cps/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
http://jenkins-ci.org/commit/workflow-plugin/5b102a5b06745b1eeeb7c1305096bf80e37c1a90
Log:
Merge pull request #372 from jenkinsci/eachClosure-JENKINS-26481

JENKINS-26481 JENKINS-27421 Fix ArrayList$Itr, and integration test for Object.each(Closure)

Compare: https://github.com/jenkinsci/workflow-plugin/compare/e3906483924d...5b102a5b0674

scm_issue_link@java.net (JIRA)

unread,
Apr 5, 2016, 4:31:17 PM4/5/16
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Jesse Glick
Path:

aggregator/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
http://jenkins-ci.org/commit/workflow-cps-plugin/0ceee9b7cb7bded004c2222d93e2927230b5d9c5
Log:
JENKINS-27421 Reproduced in test.
Originally-Committed-As: e916b5e9f35b00db679360dc29e42658d566efb7

scm_issue_link@java.net (JIRA)

unread,
Apr 5, 2016, 4:33:24 PM4/5/16
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Jesse Glick
Path:

cps/src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java
cps/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
http://jenkins-ci.org/commit/workflow-cps-plugin/4ab54f1ad2ef9685e66804dc3d3de7add29fc505


Log:
[FIXED JENKINS-27421] Producing a safe serialization replacement for ArrayList$Itr,

Originally-Committed-As: a990468a828e64961eda24332673c9cb8835c629

scm_issue_link@java.net (JIRA)

unread,
Apr 5, 2016, 4:33:25 PM4/5/16
to jenkinsc...@googlegroups.com
SCM/JIRA link daemon resolved as Fixed
Status: In Progress Resolved
Resolution: Fixed

jglick@cloudbees.com (JIRA)

unread,
May 11, 2016, 4:51:08 PM5/11/16
to jenkinsc...@googlegroups.com
Jesse Glick reopened an issue
 

Best to reopen since the original fix covered only ArrayList, not other collections.

Change By: Jesse Glick
Resolution: Fixed
Status: Resolved Reopened
Reply all
Reply to author
Forward
0 new messages