[JIRA] [remoting] (JENKINS-23271) Intermittent Invalid Object ID in remoting module

Change By:	Oleg Nenashev
Status:	Open In Progress

This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)

o.v.nenashev@gmail.com (JIRA)

unread,

Aug 5, 2016, 6:09:05 AM8/5/16

to jenkinsc...@googlegroups.com

Oleg Nenashev commented on

Re: Intermittent Invalid Object ID in remoting module

I'm currently digging into the issue.
Some analysis is available here (in progress): https://docs.google.com/document/d/1nu7jD2bNqEzD7kOM1R0VUpExycPNUq8wOTTLrr9xvOY/edit

o.v.nenashev@gmail.com (JIRA)

unread,

Aug 5, 2016, 7:44:04 AM8/5/16

to jenkinsc...@googlegroups.com

Oleg Nenashev commented on

Re: Intermittent Invalid Object ID in remoting module

I suspect an automatic object unexporter glitch.

Major changes in it:

Jun 10, 2015: New implementation of Unexporter
- https://github.com/jenkinsci/remoting/commit/7e858e3651f1e535d289eebaa92084c31a4b610b
- Released at: remoting-2.52 (jenkins-1.621+, LTS - 1.609.3)
- The issue is reported before and after this change
Apr 20, 2016 - Ensure that the unexporter cleans up whatever шt can each sweep
- https://github.com/jenkinsci/remoting/commit/f4d287652e09da627f51779bec1c3ca7f1d46e70
- Released at: remoting-2.58 (jenkins-2.4,LTS- 2.7.1+)

2.58 may have occasionally fixed the issue, I'm investigating it now.
Has anybody seen the issue on Jenkins 2.4+ including 2.7.1 or 2.7.2?

o.v.nenashev@gmail.com (JIRA)

unread,

Aug 7, 2016, 12:54:09 PM8/7/16

to jenkinsc...@googlegroups.com

Oleg Nenashev commented on

Re: Intermittent Invalid Object ID in remoting module

> Has anybody seen the issue on Jenkins 2.4+ including 2.7.1 or 2.7.2?

NVM, the bug still exist there

o.v.nenashev@gmail.com (JIRA)

unread,

Aug 10, 2016, 2:51:03 PM8/10/16

to jenkinsc...@googlegroups.com

Oleg Nenashev commented on

Re: Intermittent Invalid Object ID in remoting module

Created https://github.com/jenkinsci/remoting/pull/99 and https://github.com/jenkinsci/jenkins/pull/2503, which seem to solve/suppress issue in my case. Need a code review BTW

hai_lh@126.com (JIRA)

unread,

Aug 29, 2016, 3:49:06 AM8/29/16

to jenkinsc...@googlegroups.com

hai li commented on

Re: Intermittent Invalid Object ID in remoting module

Any update on this? Thanks!

mmitche@microsoft.com (JIRA)

unread,

Sep 6, 2016, 7:02:07 PM9/6/16

to jenkinsc...@googlegroups.com

Matthew Mitchell commented on

Re: Intermittent Invalid Object ID in remoting module

So the proposed fix isn't the correct one?

michael.yavorovsky@gmail.com (JIRA)

unread,

Sep 30, 2016, 5:25:14 AM9/30/16

to jenkinsc...@googlegroups.com

Michael Yavorovsky updated

Jenkins /

Change By:	Michael Yavorovsky
Status:	In Progress Review

michael.yavorovsky@gmail.com (JIRA)

unread,

Sep 30, 2016, 5:25:23 AM9/30/16

to jenkinsc...@googlegroups.com

Michael Yavorovsky updated

Jenkins /

Change By:	Michael Yavorovsky
Status:	In Review Progress

o.v.nenashev@gmail.com (JIRA)

unread,

Sep 30, 2016, 6:26:08 AM9/30/16

to jenkinsc...@googlegroups.com

Oleg Nenashev commented on

Re: Intermittent Invalid Object ID in remoting module

Not correct. Maybe I'll apply a hack in Jenkins core, which should at least reduce the impact

thefriendlycoder@gmail.com (JIRA)

unread,

Oct 14, 2016, 12:29:03 PM10/14/16

to jenkinsc...@googlegroups.com

Kevin Phillips commented on

Re: Intermittent Invalid Object ID in remoting module

It would appear as though we're hitting this took and it would appear that the errors are correlated to a recent Java update on our build farm as well, but we updated from an older build of 1.7 (update 40 iirc) to a newer build of 1.7 (update 50 iirc), so maybe some change between those versions is somehow contributing to the problem.

Also, in our case we see the errors happen when running builds within docker containers, and the errors occur as the docker image is being downloaded to the agent and / or when the container is being launched (ie: before any of the actual build operations take place).

So, given the fact that this issue has been outstanding for years and appears (based on a review of the comment thread) that it is not being considered priority for anyone to fix any time soon, are there any suggestions to mitigate, workaround or otherwise avoid this issue in the meantime?

thefriendlycoder@gmail.com (JIRA)

unread,

Oct 14, 2016, 12:31:03 PM10/14/16

to jenkinsc...@googlegroups.com

Kevin Phillips edited a comment on

Re: Intermittent Invalid Object ID in remoting module

It would appear as though we're hitting this took too and it would appear that the errors are correlated to a recent Java update on our build farm as well, but we updated from an older build of 1.7 (update 40 iirc) to a newer build of 1.7 (update 50 iirc) and at no point were using a 1.8 version , so maybe some change between those versions update 40 and 50 is somehow contributing to the problem. Not sure if this helps further isolate the issue or not.

Also, in our case we see the errors happen when running builds within docker containers, and the errors occur as the docker image is being downloaded to the agent and / or when the container is being launched (ie: before any of the actual build operations take place).

So, given the fact that this issue has been outstanding for years and appears (based on a review of the comment thread) that it is not being considered priority for anyone to fix any time soon, are there any suggestions to mitigate, workaround or otherwise avoid this issue in the meantime?

doz10us@gmail.com (JIRA)

unread,

Oct 14, 2016, 12:36:02 PM10/14/16

to jenkinsc...@googlegroups.com

Kirill Bucharsky commented on

Re: Intermittent Invalid Object ID in remoting module

Kevin Phillips, I noted in one environment that issue tends to happen more often if you have more simultaneous builds/configurations. So reducing from few dozens concurrent to about a dozen helped a bit.
It was like 3-5 failed builds of 100 and lowered to 1 per 200-300.

bfurtado@apple.com (JIRA)

unread,

Oct 14, 2016, 1:12:08 PM10/14/16

to jenkinsc...@googlegroups.com

Ben Furtado updated an issue

Jenkins /

Change By:	Ben Furtado
Comment:	Getting the same error on 1.632. Any ETA for resolution?

john.lengeling@gmail.com (JIRA)

unread,

Oct 14, 2016, 1:39:03 PM10/14/16

to jenkinsc...@googlegroups.com

John Lengeling commented on

Re: Intermittent Invalid Object ID in remoting module

We haven't seen the issue on any of our 10+ jenkins severs since we started running Java 1.7.80.
Would be interesting if you see the problem under that version of java.

o.v.nenashev@gmail.com (JIRA)

unread,

Oct 14, 2016, 4:43:08 PM10/14/16

to jenkinsc...@googlegroups.com

Oleg Nenashev commented on

Re: Intermittent Invalid Object ID in remoting module

John Lengeling Java 7 has a much lower chance of such issue

scm_issue_link@java.net (JIRA)

unread,

Nov 9, 2016, 6:01:10 PM11/9/16

to jenkinsc...@googlegroups.com

SCM/JIRA link daemon commented on

Re: Intermittent Invalid Object ID in remoting module

Code changed in jenkins
User: Bertrand Roussel
Path:
client/pom.xml
http://jenkins-ci.org/commit/swarm-plugin/8ad209aaafad5ec6256aa547702870da083ff6cd
Log:
Upgrade remoting API to 2.62

Version 2.62 contains a fix for the following bugs:

This occurs mostly with Java 8.

The remoting version matches with Jenkins 2.18/2.19/2.19.1/2.19.2.

o.v.nenashev@gmail.com (JIRA)

unread,

Nov 14, 2016, 7:57:11 PM11/14/16

to jenkinsc...@googlegroups.com

Oleg Nenashev commented on

Re: Intermittent Invalid Object ID in remoting module

Created a PR, which seems to be a plausible workaround.
https://github.com/jenkinsci/jenkins/pull/2635

o.v.nenashev@gmail.com (JIRA)

unread,

Nov 27, 2016, 1:23:09 PM11/27/16

to jenkinsc...@googlegroups.com

Oleg Nenashev resolved as Fixed

Marking as Resolved for a while, it should become available in 2.32.1 LTS. I have found no evidence of other cases in the Jenkins core, which cause such object de-provisioning (but they may exist). Please create an issue if it happens in 2.31+ on your instances

Jenkins /

Change By:	Oleg Nenashev
Status:	In Progress Resolved
Resolution:	Fixed

Kuypers.Dirk@googlemail.com (JIRA)

unread,

Nov 29, 2016, 8:18:06 AM11/29/16

to jenkinsc...@googlegroups.com

Dirk Kuypers commented on

Re: Intermittent Invalid Object ID in remoting module

Oleg, I am still seeing these issues with 2.34:

FATAL: java.util.concurrent.ExecutionException: Invalid object ID 811 iota=812
hudson.remoting.RemotingSystemException: java.util.concurrent.ExecutionException: Invalid object ID 811 iota=812
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:272)
at com.sun.proxy.$Proxy85.join(Unknown Source)
at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1009)
at hudson.plugins.clearcase.HudsonClearToolLauncher.run(HudsonClearToolLauncher.java:143)
at hudson.plugins.clearcase.ClearToolExec.describe(ClearToolExec.java:126)
at hudson.plugins.clearcase.base.BaseHistoryAction.parseUpdt(BaseHistoryAction.java:137)
at hudson.plugins.clearcase.base.BaseHistoryAction.runLsHistory(BaseHistoryAction.java:156)
at hudson.plugins.clearcase.history.AbstractHistoryAction.runAndFilterLsHistory(AbstractHistoryAction.java:259)
at hudson.plugins.clearcase.history.AbstractHistoryAction.getChanges(AbstractHistoryAction.java:81)
at hudson.plugins.clearcase.AbstractClearCaseScm.saveChangeLog(AbstractClearCaseScm.java:1106)
at hudson.plugins.clearcase.AbstractClearCaseScm.checkout(AbstractClearCaseScm.java:395)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1278)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
at hudson.model.Run.execute(Run.java:1728)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:404)
Caused by: java.util.concurrent.ExecutionException: Invalid object ID 811 iota=812
at hudson.remoting.ExportTable.diagnoseInvalidObjectId(ExportTable.java:408)
at hudson.remoting.ExportTable.get(ExportTable.java:333)
at hudson.remoting.Channel.getExportedObject(Channel.java:660)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:885)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:869)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:828)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:336)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:94)
at java.lang.Thread.run(Unknown Source)
at ......remote call to Channel to /10.0.50.50(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1537)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
at hudson.remoting.Channel.call(Channel.java:822)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:256)
... 18 more

Caused by: java.lang.Exception: Object was recently deallocated

#811 (ref.0) : object=null type=hudson.Launcher$RemoteLaunchCallable$1 interfaces=[hudson.Launcher$RemoteProcess]
Created at Mon Nov 28 20:32:47 CET 2016
at hudson.remoting.ExportTable$Entry.<init>(ExportTable.java:102)
at hudson.remoting.ExportTable.export(ExportTable.java:308)
at hudson.remoting.Channel.internalExport(Channel.java:656)
at hudson.remoting.Channel.export(Channel.java:647)
at hudson.remoting.Channel.export(Channel.java:617)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1168)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1131)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:336)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:94)
at java.lang.Thread.run(Unknown Source)
Released at Mon Nov 28 20:32:49 CET 2016
at hudson.remoting.ExportTable$Entry.release(ExportTable.java:134)
at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:449)
at hudson.remoting.Channel.unexport(Channel.java:684)
at hudson.remoting.Channel.unexport(Channel.java:673)
at hudson.remoting.UnexportCommand.execute(UnexportCommand.java:43)
at hudson.remoting.Channel$1.handle(Channel.java:526)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:83)
Caused by: Command hudson.remoting.UnexportCommand@154ee63 created at
at hudson.remoting.Command.<init>(Command.java:70)
at hudson.remoting.Command.<init>(Command.java:53)

at hudson.remoting.UnexportCommand.<init>(UnexportCommand.java:33)

at hudson.remoting.RemoteInvocationHandler$PhantomReferenceImpl.cleanup(RemoteInvocationHandler.java:366)
at hudson.remoting.RemoteInvocationHandler$PhantomReferenceImpl.access$900(RemoteInvocationHandler.java:325)
at hudson.remoting.RemoteInvocationHandler$Unexporter.run(RemoteInvocationHandler.java:583)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.Exception: Proxy hudson.remoting.RemoteInvocationHandler@32b was created for interface hudson.Launcher$RemoteProcess
at hudson.remoting.RemoteInvocationHandler.<init>(RemoteInvocationHandler.java:126)
at hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:138)
at hudson.remoting.Channel.export(Channel.java:648)
at hudson.remoting.Channel.export(Channel.java:617)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1168)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1131)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:336)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:94)
at java.lang.Thread.run(Unknown Source)
at hudson.remoting.ExportTable.diagnoseInvalidObjectId(ExportTable.java:401)
at hudson.remoting.ExportTable.get(ExportTable.java:333)
at hudson.remoting.Channel.getExportedObject(Channel.java:660)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:885)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:869)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:828)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:336)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:94)
at java.lang.Thread.run(Unknown Source)
Caused by: Released at Mon Nov 28 20:32:49 CET 2016
at hudson.remoting.ExportTable$Entry.release(ExportTable.java:134)
at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:449)
at hudson.remoting.Channel.unexport(Channel.java:684)
at hudson.remoting.Channel.unexport(Channel.java:673)
at hudson.remoting.UnexportCommand.execute(UnexportCommand.java:43)
at hudson.remoting.Channel$1.handle(Channel.java:526)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:83)
Caused by: Command hudson.remoting.UnexportCommand@33cebea8 created at
at hudson.remoting.Command.<init>(Command.java:70)
at hudson.remoting.Command.<init>(Command.java:53)

at hudson.remoting.UnexportCommand.<init>(UnexportCommand.java:33)

at hudson.remoting.RemoteInvocationHandler$PhantomReferenceImpl.cleanup(RemoteInvocationHandler.java:366)
at hudson.remoting.RemoteInvocationHandler$PhantomReferenceImpl.access$900(RemoteInvocationHandler.java:325)
at hudson.remoting.RemoteInvocationHandler$Unexporter.run(RemoteInvocationHandler.java:583)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.Exception: Proxy hudson.remoting.RemoteInvocationHandler@32b was created for interface hudson.Launcher$RemoteProcess
at hudson.remoting.RemoteInvocationHandler.<init>(RemoteInvocationHandler.java:126)
at hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:138)
at hudson.remoting.Channel.export(Channel.java:648)
at hudson.remoting.Channel.export(Channel.java:617)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1168)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1131)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:336)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:94)
at java.lang.Thread.run(Unknown Source)

I had this with three different builds last night. As these builds run only once per night they accumulate a lot of changes. And while ClearCase is accumulating these changes on the view update the job chokes with this error report. Or is this completely unrelated?

o.v.nenashev@gmail.com (JIRA)

unread,

Nov 29, 2016, 9:00:45 AM11/29/16

to jenkinsc...@googlegroups.com

Oleg Nenashev reopened an issue

Yes, it is the same issue, but with another code. Likely we need a better solution

Jenkins /

Change By:	Oleg Nenashev
Resolution:	Fixed
Status:	Resolved Reopened

Kuypers.Dirk@googlemail.com (JIRA)

unread,

Nov 29, 2016, 9:04:04 AM11/29/16

to jenkinsc...@googlegroups.com

Dirk Kuypers commented on

Re: Intermittent Invalid Object ID in remoting module

Could I help out with something where more logging is enabled? I have about 10 of these SonarQube Jobs running every night and at least one of them fails almost every night.

o.v.nenashev@gmail.com (JIRA)

unread,

Nov 29, 2016, 10:03:04 AM11/29/16

to jenkinsc...@googlegroups.com

Oleg Nenashev commented on

Re: Intermittent Invalid Object ID in remoting module

Dirk Kuypers Give me several hours. I will create a pull request with a wider solution. Then you will be able to take a PoC from the CI just to verify if it helps

o.v.nenashev@gmail.com (JIRA)

unread,

Nov 29, 2016, 5:04:03 PM11/29/16

to jenkinsc...@googlegroups.com

Oleg Nenashev commented on

Re: Intermittent Invalid Object ID in remoting module

Created https://github.com/jenkinsci/jenkins/pull/2653

thefriendlycoder@gmail.com (JIRA)

unread,

Dec 1, 2016, 8:26:05 AM12/1/16

to jenkinsc...@googlegroups.com

Kevin Phillips commented on

Re: Intermittent Invalid Object ID in remoting module

We have been seeing this error periodically as well.

Just out of curiosity, is there any indication on which version(s) of Jenkins (ie: v1 / v2) the fix will be applied once one has been approved?

o.v.nenashev@gmail.com (JIRA)

unread,

Dec 1, 2016, 10:29:09 AM12/1/16

to jenkinsc...@googlegroups.com

Oleg Nenashev commented on

Re: Intermittent Invalid Object ID in remoting module

Jenkins 1.x is not supported by the community anymore (though I'm not a fan of the current release model).
The fix will be integrated into 2.35 and likely then backported to 2.32.1

scm_issue_link@java.net (JIRA)

unread,

Dec 3, 2016, 10:56:03 AM12/3/16

to jenkinsc...@googlegroups.com

SCM/JIRA link daemon commented on

Re: Intermittent Invalid Object ID in remoting module

Code changed in jenkins
User: Oleg Nenashev
Path:
core/src/main/java/hudson/Launcher.java
core/src/main/java/hudson/Proc.java
http://jenkins-ci.org/commit/jenkins/29893354a52efac20efcb4e0e91723f3bd889059
Log:
JENKINS-23271 - Process statuses of Remote process join() operations directly inside methods (#2653)

JENKINS-23271 - Process statuses of Remote process join() operations directly inside methods

JENKINS-23271 - Also prevent the issue when the kill() command is the last call in the usage sequence

andrew.paul.gray@gmail.com (JIRA)

unread,

Dec 6, 2016, 9:48:07 PM12/6/16

to jenkinsc...@googlegroups.com

Andrew Gray commented on

Re: Intermittent Invalid Object ID in remoting module

Should this now be resolved with release of 2.35?

o.v.nenashev@gmail.com (JIRA)

unread,

Dec 7, 2016, 3:11:07 AM12/7/16

to jenkinsc...@googlegroups.com

Oleg Nenashev updated an issue

Jenkins /

Change By:	Oleg Nenashev
Labels:	lts-candidate remoting

o.v.nenashev@gmail.com (JIRA)

unread,

Dec 7, 2016, 3:12:08 AM12/7/16

to jenkinsc...@googlegroups.com

Oleg Nenashev resolved as Fixed

At least more failure cases should be solved

Jenkins /

Change By:	Oleg Nenashev
Status:	Reopened Resolved
Resolution:	Fixed

ogondza@gmail.com (JIRA)

unread,

Dec 8, 2016, 7:51:04 AM12/8/16

to jenkinsc...@googlegroups.com

Oliver Gondža updated an issue

Jenkins /

Change By:	Oliver Gondža
Labels:	lts 2.32.1 - candidate fixed remoting

scm_issue_link@java.net (JIRA)

unread,

Dec 8, 2016, 8:29:10 AM12/8/16

to jenkinsc...@googlegroups.com

SCM/JIRA link daemon commented on

Re: Intermittent Invalid Object ID in remoting module

Code changed in jenkins
User: Oleg Nenashev
Path:
core/src/main/java/hudson/Launcher.java

http://jenkins-ci.org/commit/jenkins/fcfd271ad5ad5947075052d37177249c760f6184
Log:
~~JENKINS-23271~~ - Prevent early deallocation of the Proc instance by GC in ProcStarter#join() (#2635)

~~JENKINS-23271~~ - Prevent the prelimimary deallocation of the Proc instance by GC

It is a hackish way, which likely prevents a preliminary deallocation of the spawned RemoteProc instance, which we see in ~~JENKINS-23271~~. Proc instance was not actually required in the original code since we were creating and using RemoteInvocationHandler wrapper only, and the theory discussed with @stephenc was that object gets removed by Java8 garbage collector before we get into join().

This fix enforces the persistency of ProcStarter#start() result by adding logging and the enforced volatile field (maybe the last one is not really required, but JIT compiler in Java implementations may be smart enough to skip unused loggers)

This is a pretty old fix from August, which has been soak tested on my instance for several weeks (mid-August => Jenkins World). On the reference instance (just a small Jenkins instance with 4 agents and very frequent builds with CommandInterpreter steps) I saw 2 failures over the period. On the fixed instance - 0. It does not proof anything, but at least the fix was soak tested a bit

~~JENKINS-23271~~ - Get rid of the procHolderForJoin field

~~JENKINS-23271~~ - Also put the check into the finally statement as @stephenc proposed

Remove assert

(cherry picked from commit fd6c6aff929be9818f4eb4b84ed6b4593356853f)

scm_issue_link@java.net (JIRA)

unread,

Dec 8, 2016, 8:29:30 AM12/8/16

to jenkinsc...@googlegroups.com

SCM/JIRA link daemon commented on

Re: Intermittent Invalid Object ID in remoting module

Code changed in jenkins
User: Oleg Nenashev
Path:
core/src/main/java/hudson/Launcher.java

core/src/main/java/hudson/Proc.java
http://jenkins-ci.org/commit/jenkins/5500e5c00bc19e6c421c9aaa88f6dbb57c666db3

Log:
~~JENKINS-23271~~ - Process statuses of Remote process join() operations directly inside methods (#2653)

~~JENKINS-23271~~ - Process statuses of Remote process join() operations directly inside methods

~~JENKINS-23271~~ - Also prevent the issue when the kill() command is the last call in the usage sequence

(cherry picked from commit 29893354a52efac20efcb4e0e91723f3bd889059)

Compare: https://github.com/jenkinsci/jenkins/compare/dacbe03e0d4e...5500e5c00bc1

scm_issue_link@java.net (JIRA)

unread,

Feb 13, 2017, 12:17:13 PM2/13/17

to jenkinsc...@googlegroups.com

SCM/JIRA link daemon commented on

Re: Intermittent Invalid Object ID in remoting module

Code changed in jenkins
User: Oleg Nenashev
Path:

docs/configuration.md
src/main/java/hudson/remoting/Base64.java
src/main/java/hudson/remoting/Channel.java
src/main/java/hudson/remoting/ErrorPropagatingOutputStream.java
src/main/java/hudson/remoting/ExportTable.java
src/main/java/hudson/remoting/LocalChannel.java
src/main/java/hudson/remoting/RemoteInvocationHandler.java
src/main/java/hudson/remoting/VirtualChannel.java
http://jenkins-ci.org/commit/remoting/27bb51dd14db44952348539d15ee004c90c248b2
Log:
Document and annotate object Export/Unexport logic

During last releases we have fixed several issues related to the object export/unexport operations (~~JENKINS-23271~~, ~~JENKINS-41852~~). I decided that it makes sense to improve the method documentation just to make Javadocs more explicit.

The change also adds missing documentation of hudson.remoting.ExportTable.unexportLogSize