ConcurrentModificationException saving artifacts to GCP
13 views
Skip to first unread message
kuisathaverat
Oct 28, 2020, 5:34:59 PM10/28/20
to jenkins...@googlegroups.com
Hi,
We are facing some issues with a job that has about 100 stages in parallel, we track down the issue and it happens when we storage artifacts on GCP in parallel, the thing is that I am not sure it is related to the plugin directly, Could someone take a look to this stack trace and confirm that it is not related to the Core? BTW we are using Jenkins 2.252
19:12:54 java.util.ConcurrentModificationException 19:12:54 at java.util.HashMap$HashIterator.nextNode(HashMap.java:1445) 19:12:54 at java.util.HashMap$KeyIterator.next(HashMap.java:1469) 19:12:54 at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:73) 19:12:54 at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) 19:12:54 at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) 19:12:54 at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84) 19:12:54 at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:264) 19:12:54 at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:251) 19:12:54 Caused: java.lang.RuntimeException: Failed to serialize com.google.jenkins.plugins.storage.reports.BuildGcsUploadReport#files for class com.google.jenkins.plugins.storage.reports.BuildGcsUploadReport 19:12:54 at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:255) 19:12:54 at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:223) 19:12:54 at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) 19:12:54 at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209) 19:12:54 at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150) 19:12:54 at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) 19:12:54 at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) 19:12:54 at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) 19:12:54 at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88) 19:12:54 at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:64) 19:12:54 at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:74) 19:12:54 at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) 19:12:54 at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) 19:12:54 at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84) 19:12:54 at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:264) 19:12:54 at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:251) 19:12:54 Caused: java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class org.jenkinsci.plugins.workflow.job.WorkflowRun 19:12:54 at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:255) 19:12:54 at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:223) 19:12:54 at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) 19:12:54 at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209) 19:12:54 at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150) 19:12:54 at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) 19:12:54 at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) 19:12:54 at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) 19:12:54 at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82) 19:12:54 at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37) 19:12:54 at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026) 19:12:54 at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015) 19:12:54 at com.thoughtworks.xstream.XStream.toXML(XStream.java:988) 19:12:54 at hudson.util.XStream2.toXMLUTF8(XStream2.java:313) 19:12:54 at org.jenkinsci.plugins.workflow.support.PipelineIOUtils.writeByXStream(PipelineIOUtils.java:34) 19:12:54 at org.jenkinsci.plugins.workflow.job.WorkflowRun.save(WorkflowRun.java:1199) 19:12:54 at hudson.BulkChange.commit(BulkChange.java:98) 19:12:54 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1475) 19:12:54 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:489) 19:12:54 at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:38) 19:12:54 at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:136) 19:12:54 at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) 19:12:54 at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) 19:12:54 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 19:12:54 at java.util.concurrent.FutureTask.run(FutureTask.java:266) 19:12:54 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 19:12:54 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 19:12:54 at java.lang.Thread.run(Thread.java:748)
Thanks
Ivan Fernandez
Ivan Fernandez Calvo
Oct 29, 2020, 11:53:38 AM10/29/20
to Jenkins Users
We have workaround the issue by using a semaphore to call the GCP storage step
import groovy.transform.Field
@Field def lock = false
@NonCPS
def isLock(){
synchronized(lock) {
return lock
}
}
def waitForUnlock(){
while(isLock()){
sleep 10
}
}
def uploadPackages(bucketUri, baseDir){
waitForUnlock()
setLock()
googleStorageUpload(bucket: bucketUri,
credentialsId: "${JOB_GCS_CREDENTIALS}",
pathPrefix: "${baseDir}/build/distributions/",
pattern: "${baseDir}/build/distributions/**/*",
sharedPublicly: true,
showInline: true
)
setUnlock()
}
import groovy.transform.Field
@Field def lock = false
@NonCPS
def setLock(){
synchronized(lock) {
lock = true
}
}
@NonCPS
def setUnlock(){
synchronized(lock) {
lock = false
}
}
def setLock(){
synchronized(lock) {
lock = true
}
}
@NonCPS
def setUnlock(){
synchronized(lock) {
lock = false
}
}
@NonCPS
def isLock(){
synchronized(lock) {
return lock
}
}
def waitForUnlock(){
while(isLock()){
sleep 10
}
}
def uploadPackages(bucketUri, baseDir){
waitForUnlock()
setLock()
googleStorageUpload(bucket: bucketUri,
credentialsId: "${JOB_GCS_CREDENTIALS}",
pathPrefix: "${baseDir}/build/distributions/",
pattern: "${baseDir}/build/distributions/**/*",
sharedPublicly: true,
showInline: true
)
setUnlock()
}
Ivan Fernandez Calvo
Oct 29, 2020, 4:44:07 PM10/29/20
to Jenkins Users
Well, we really did not workaround the issue, it simply happens fewer times, the real solution will be https://github.com/jenkinsci/google-storage-plugin/pull/120
Reply all
Reply to author
Forward
0 new messages