[JIRA] (JENKINS-57553) High throughput causes ConcurrentModificationException

11 views
Skip to first unread message

jieshe@microsoft.com (JIRA)

unread,
May 20, 2019, 3:03:03 AM5/20/19
to jenkinsc...@googlegroups.com
Jie Shen created an issue
 
Jenkins / Bug JENKINS-57553
High throughput causes ConcurrentModificationException
Issue Type: Bug Bug
Assignee: Azure DevOps
Components: windows-azure-storage-plugin
Created: 2019-05-20 07:02
Priority: Blocker Blocker
Reporter: Jie Shen

It looks like some kind of serialization operation is happening to an AzureBlobAction while the individualBlobs list is being modified. We have this happening several times a day at the moment, but we have a pretty high throughput of builds which is probably related.

 

hudson.remoting.ProxyException: java.util.ConcurrentModificationException       at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)…
Caused: hudson.remoting.ProxyException: java.lang.RuntimeException: Failed to serialize com.microsoftopentechnologies.windowsazurestorage.AzureBlobAction#individualBlobs for class com.microsoftopentechnologies.windowsazurestorage.AzureBlobAction       
at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)…
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

jieshe@microsoft.com (JIRA)

unread,
May 20, 2019, 3:11:02 AM5/20/19
to jenkinsc...@googlegroups.com
Jie Shen assigned an issue to Jie Shen
Change By: Jie Shen
Assignee: Azure DevOps Jie Shen

jieshe@microsoft.com (JIRA)

unread,
May 20, 2019, 3:12:01 AM5/20/19
to jenkinsc...@googlegroups.com
Jie Shen commented on Bug JENKINS-57553
 
Re: High throughput causes ConcurrentModificationException

Change back to use storage sdk should be able to fix this.

thomas@nodeable.io (JIRA)

unread,
Jul 4, 2019, 6:59:02 AM7/4/19
to jenkinsc...@googlegroups.com

I am getting the same issue at the end of a piplene that uploads a large number of individual files to Azure Storage: 

hudson.remoting.ProxyException: java.util.ConcurrentModificationException
{{ at java.util.ArrayList$Itr.checkForComodification(Unknown Source)}}
{{ at java.util.ArrayList$Itr.next(Unknown Source)}}
{{ at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:73)}}
{{ at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)}}
{{ at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)}}
{{ at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)}}
{{ at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88)}}
{{ at com.thoughtworks.xstream.converters.reflection.SerializableConverter$1.defaultWriteObject(SerializableConverter.java:214)}}
{{ at com.thoughtworks.xstream.core.util.CustomObjectOutputStream.defaultWriteObject(CustomObjectOutputStream.java:80)}}
{{ at java.util.Collections$SynchronizedCollection.writeObject(Unknown Source)}}
{{ at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}}
{{ at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)}}
{{ at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)}}
{{ at java.lang.reflect.Method.invoke(Unknown Source)}}
{{ at com.thoughtworks.xstream.converters.reflection.SerializationMethodInvoker.callWriteObject(SerializationMethodInvoker.java:135)}}
Caused: hudson.remoting.ProxyException: com.thoughtworks.xstream.converters.ConversionException: Could not call java.util.Collections$SynchronizedList.writeObject() : null
---- Debugging information ----
message : Could not call java.util.Collections$SynchronizedList.writeObject()
cause-exception : java.util.ConcurrentModificationException
cause-message : null
-------------------------------
{{ at com.thoughtworks.xstream.converters.reflection.SerializationMethodInvoker.callWriteObject(SerializationMethodInvoker.java:141)}}
{{ at com.thoughtworks.xstream.converters.reflection.SerializableConverter.doMarshal(SerializableConverter.java:259)}}
{{ at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.marshal(AbstractReflectionConverter.java:83)}}
{{ at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)}}
{{ at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)}}
{{ at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)}}
{{ at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88)}}
{{ at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.marshal(AbstractReflectionConverter.java:81)}}
{{ at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)}}
{{ at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)}}
{{ at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)}}
{{ at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:263)}}
{{ at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:250)}}


Caused: hudson.remoting.ProxyException: java.lang.RuntimeException: Failed to serialize com.microsoftopentechnologies.windowsazurestorage.AzureBlobAction#individualBlobs for class com.microsoftopentechnologies.windowsazurestorage.AzureBlobAction

{{ at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:254)}}
{{ at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:222)}}
{{ at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)}}
{{ at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:208)}}
{{ at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:149)}}
{{ at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)}}
{{ at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)}}
{{ at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)}}
{{ at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88)}}
{{ at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:64)}}
{{ at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:74)}}
{{ at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)}}
{{ at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)}}
{{ at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)}}
{{ at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:263)}}
{{ at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:250)}}
Caused: hudson.remoting.ProxyException: java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class org.jenkinsci.plugins.workflow.job.WorkflowRun
{{ at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:254)}}
{{ at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:222)}}
{{ at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)}}
{{ at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:208)}}
{{ at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:149)}}
{{ at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)}}
{{ at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)}}
{{ at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)}}
{{ at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82)}}
{{ at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)}}
{{ at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026)}}
{{ at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015)}}
{{ at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)}}
{{ at hudson.util.XStream2.toXMLUTF8(XStream2.java:313)}}
{{ at org.jenkinsci.plugins.workflow.support.PipelineIOUtils.writeByXStream(PipelineIOUtils.java:34)}}
{{ at org.jenkinsci.plugins.workflow.job.WorkflowRun.save(WorkflowRun.java:1140)}}
{{ at hudson.BulkChange.commit(BulkChange.java:98)}}
{{ at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1476)}}
{{ at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:458)}}
{{ at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)}}
{{ at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)}}
{{ at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)}}
{{ at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)}}
{{ at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)}}
{{ at java.util.concurrent.FutureTask.run(Unknown Source)}}
{{ at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)}}
{{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)}}
{{ at java.lang.Thread.run(Unknown Source)}}

jieshe@microsoft.com (JIRA)

unread,
Jul 5, 2019, 4:03:01 AM7/5/19
to jenkinsc...@googlegroups.com
Jie Shen commented on Bug JENKINS-57553

Thomas Hansen Does this affect uploading? Does it mark the job failed?

thomas@nodeable.io (JIRA)

unread,
Jul 5, 2019, 4:14:02 AM7/5/19
to jenkinsc...@googlegroups.com

It does not seem to affect uploading, but it does end up marking the job as a failure.

When the same pipeline runs, but with just a small number of files to upload, then I don't get this error and the job completes successfully. 

 

jieshe@microsoft.com (JIRA)

unread,
Jul 8, 2019, 1:51:02 AM7/8/19
to jenkinsc...@googlegroups.com
Jie Shen resolved as Fixed
 
High throughput causes ConcurrentModificationException
Change By: Jie Shen
Status: Open Resolved
Resolution: Fixed
Released As: 1.0.2
Reply all
Reply to author
Forward
0 new messages