[JIRA] (JENKINS-58068) Races / corruption on revision.txt in pipeline-parallel

10 views
Skip to first unread message

andreas.ringlstetter@gmail.com (JIRA)

unread,
Jun 18, 2019, 8:45:02 AM6/18/19
to jenkinsc...@googlegroups.com
Andreas Ringlstetter created an issue
 
Jenkins / Bug JENKINS-58068
Races / corruption on revision.txt in pipeline-parallel
Issue Type: Bug Bug
Assignee: Ivan Fernandez Calvo
Components: subversion-plugin
Created: 2019-06-18 12:44
Priority: Major Major
Reporter: Andreas Ringlstetter

In the scope of JENKINS-39968, the subversion plugin suffers from additional races due to the dependency on the "revision.txt" file as part of the checkout and change log computation process.

Unsynchronized access to "revision.txt" causes `SubversionChangeLogBuilder ` to fail spuriously, as the file may be written to concurrently by other running checkout processes.

The final "revision.txt" eventually reaches a stable state where it only contains the revision of the last "checkout" step, but in the process `SubversionChangeLogBuilder` has failed to record changes repeatedly (bailout with "No revision found for ....") and the "build.xml" is corrupted and contains fragments like the following:

    <org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState plugin="workflow-scm-step@2.9">
      <revisionStates>
        <entry>
          <string>svn http://redacted/trunk</string>
          <hudson.scm.SVNRevisionState plugin="subve...@2.12.1">
            <revisions>
              <entry>
                <string>ed/trunk</string>
                <long>16122</long>
              </entry>
              <entry>
                <string>svn http://redacted/trunk</string>
                <long>12362</long>
              </entry>
            </revisions>
          </hudson.scm.SVNRevisionState>
        </entry>
      </revisionStates>
    </org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState>

In summary:

  • Unsafe reads from "revision.txt" which contains data from parallel checkout processes
  • Lost writes to "revision.txt" from previous checkout steps in the same build, with unstable results

While producing "revision.txt" as a side product of the build process may be desired to keep as a side effect, using it internally means state between `SubversionSCM` instances in the same build is shared in an unsafe way.

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

andreas.ringlstetter@gmail.com (JIRA)

unread,
Jun 18, 2019, 8:52:02 AM6/18/19
to jenkinsc...@googlegroups.com
Andreas Ringlstetter updated an issue
In the scope of JENKINS-39968, the subversion plugin suffers from additional races due to the dependency on the "revision.txt" file as part of the checkout and change log computation process.

Unsynchronized access to "revision.txt" causes `SubversionChangeLogBuilder ` to fail spuriously, as the file may be written to concurrently by other running checkout processes.

The final "revision.txt" eventually reaches a stable state where it only contains the revision of the last "checkout" step, but in the process `SubversionChangeLogBuilder` has failed to record changes repeatedly (bailout with "No revision found for ....") and the "build.xml" is corrupted and contains fragments like the following:

{code:java}

    <org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState plugin="workflow-scm-step@2.9">
      <revisionStates>
        <entry>
          <string>svn http://redacted/trunk</string>
          <hudson.scm.SVNRevisionState plugin="subve...@2.12.1">
            <revisions>
              <entry>
                <string>ed/trunk</string>
                <long>16122</long>
              </entry>
              <entry>
                <string>svn http://redacted/trunk</string>
                <long>12362</long>
              </entry>
            </revisions>
          </hudson.scm.SVNRevisionState>
        </entry>
      </revisionStates>
    </org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState>
{code}

In summary:
* Unsafe reads from "revision.txt" which contains data from parallel checkout processes
* Lost writes to "revision.txt" from previous checkout steps in the same build, with unstable results

While producing "revision.txt" as a side product of the build process may be desired to keep as a side effect, using it internally means state between `SubversionSCM` instances in the same build is shared in an unsafe way.
Assuming it is only written to once per build may be somewhat fitting to Freestyle projects, but is unsuited for pipeline projects.

andreas.ringlstetter@gmail.com (JIRA)

unread,
Jun 18, 2019, 8:53:02 AM6/18/19
to jenkinsc...@googlegroups.com
Andreas Ringlstetter updated an issue
In the scope of JENKINS-39968 When used in a pipleine parallel block , the subversion plugin suffers from additional races raceconditions due to the dependency on the "revision.txt" file as part of the checkout and change log computation process.

kuisathaverat@gmail.com (JIRA)

unread,
Jun 18, 2019, 12:05:02 PM6/18/19
to jenkinsc...@googlegroups.com
Ivan Fernandez Calvo assigned an issue to Unassigned
Change By: Ivan Fernandez Calvo
Assignee: Ivan Fernandez Calvo

dieter.dobbelaere@excentis.com (JIRA)

unread,
Oct 1, 2019, 4:52:03 AM10/1/19
to jenkinsc...@googlegroups.com
Dieter Dobbelaere commented on Bug JENKINS-58068
 
Re: Races / corruption on revision.txt in pipeline-parallel

We have experienced the exact same issue in our parallel pipelines (we could also track it down to corruption of "revision.txt"). This issue has plagued us with lots of spurious SCM triggered builds, which were not caused by real SCM changes, but by corrupted internal state files.

Our workaround was to define a new "trigger" pipeline whose sole purpose it to check out all the sources (subversion + git repos) in a single stage. This "trigger" pipeline is triggered by SCM changes and triggers the main pipeline after completion.

This solves the issue. If possible, an update of the subversion-plugin would be most welcome though, such that this artificial construction can be avoided.

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

dieter.dobbelaere@excentis.com (JIRA)

unread,
Oct 1, 2019, 5:20:02 AM10/1/19
to jenkinsc...@googlegroups.com
Dieter Dobbelaere edited a comment on Bug JENKINS-58068
We have experienced the exact same issue in our parallel pipelines (we could also track it down to corruption of "revision.txt"). This issue has plagued us with lots of spurious SCM triggered builds, which were not caused by _real_ SCM changes, but by corrupted internal state files.


Our workaround was to define a new "trigger" pipeline whose sole purpose it to check out all the sources (subversion + git repos) in a single stage. This "trigger" pipeline is triggered by SCM changes and triggers the main pipeline after completion.

This solves the issue of spurious SCM triggered builds ("revision . txt" is still corrupted in the main pipeline folder though).

If possible, an update of the subversion-plugin would be most welcome though, such that this artificial construction can be avoided.

dieter.dobbelaere@excentis.com (JIRA)

unread,
Oct 1, 2019, 5:21:02 AM10/1/19
to jenkinsc...@googlegroups.com
Dieter Dobbelaere edited a comment on Bug JENKINS-58068
We have experienced the exact same issue in our parallel pipelines (we could also track it down to corruption of "revision.txt"). This issue has plagued us with lots of spurious SCM triggered builds, which were not caused by _real_ SCM changes, but by corrupted internal state files.

Our workaround was to define a new "trigger" pipeline whose sole purpose it to check out all the sources (subversion + git repos) in a single stage. This "trigger" pipeline is triggered by SCM changes and triggers the main pipeline after completion.

This solves the issue of spurious SCM triggered builds ("revision.txt" is and "build.xml" are still corrupted in the main pipeline folder though).


If possible, an update of the subversion-plugin would be most welcome though, such that this artificial construction can be avoided.
Reply all
Reply to author
Forward
0 new messages