[JIRA] (JENKINS-58671) Concurrency problems with SVN workflow libs => maybe upgrade MapDB

6 views
Skip to first unread message

r.fuereder@xortex.com (JIRA)

unread,
Jul 26, 2019, 4:48:04 AM7/26/19
to jenkinsc...@googlegroups.com
Reinhold Füreder created an issue
 
Jenkins / Bug JENKINS-58671
Concurrency problems with SVN workflow libs => maybe upgrade MapDB
Issue Type: Bug Bug
Assignee: Ivan Fernandez Calvo
Components: mapdb-api-plugin, subversion-plugin
Created: 2019-07-26 08:47
Environment: Everything relevant is latest and greatest; Jenkins in Docker
Priority: Minor Minor
Reporter: Reinhold Füreder

Initial Docker based Jenkins startup based on JCasC and job-dsl and pipeline jobs auto-scheduled (via job-dsl 'queue' command) right after their creation that all use a shared Jenkins pipeline library in SVN.

  • This did not hurt and led to successful pipeline job executions
    2019-07-26 08:16:12.952+0000 [id=150]   WARNING j.s.i.s.SVNRepositoryView#<init>: failing to make/load /var/jenkins_home/caches/svn-fa86427e-e420-4d14-a4fd-9ab1bdfcde57.db
    java.lang.AssertionError: unknown trans log instruction '-58' at log offset: 176
            at org.mapdb.StoreWAL.replayLogFile(StoreWAL.java:827)
            at org.mapdb.StoreWAL.commit(StoreWAL.java:606)
            at org.mapdb.EngineWrapper.commit(EngineWrapper.java:94)
            at org.mapdb.DBMaker.makeEngine(DBMaker.java:838)
            at org.mapdb.DBMaker.make(DBMaker.java:701)
            at jenkins.scm.impl.subversion.SVNRepositoryView.<init>(SVNRepositoryView.java:105)
            at jenkins.scm.impl.subversion.SubversionSCMSource.openSession(SubversionSCMSource.java:338)
            at jenkins.scm.impl.subversion.SubversionSCMSource.retrieve(SubversionSCMSource.java:294)
            at jenkins.scm.api.SCMSource.fetch(SCMSource.java:636)
            at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.lambda$retrieve$0(SCMSourceRetriever.java:89)
            at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrySCMOperation(SCMSourceRetriever.java:104)
            at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:89)
            at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:157)
            at org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:138)
            at org.jenkinsci.plugins.workflow.libs.LibraryDecorator$1.call(LibraryDecorator.java:125)
            at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1065)
            at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
            at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
            at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
            at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
            at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
            at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
            at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
            at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142)
            at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127)
            at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:561)
            at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:522)
            at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:320)
            at hudson.model.ResourceController.execute(ResourceController.java:97)
            at hudson.model.Executor.run(Executor.java:429)
    
  • This, by contrast, did hurt and led to failing builds:
    2019-07-26 08:01:00.838+0000 [id=150]   WARNING j.s.i.s.SVNRepositoryView#<init>: failing to make/load /var/jenkins_home/caches/svn-fa86427e-e420-4d14-a4fd-9ab1bdfcde57.db
    java.io.EOFException
            at org.mapdb.Volume$FileChannelVol.readFully(Volume.java:947)
            at org.mapdb.Volume$FileChannelVol.getByte(Volume.java:997)
    Caused: java.io.IOError
            at org.mapdb.Volume$FileChannelVol.getByte(Volume.java:1000)
            at org.mapdb.StoreWAL.replayLogFile(StoreWAL.java:797)
            at org.mapdb.StoreWAL.commit(StoreWAL.java:606)
            at org.mapdb.EngineWrapper.commit(EngineWrapper.java:94)
            at org.mapdb.BTreeMap.preinitCatalog(BTreeMap.java:176)
            at org.mapdb.DB.reinit(DB.java:88)
            at org.mapdb.DB.<init>(DB.java:83)
            at org.mapdb.DBMaker.make(DBMaker.java:704)
            at jenkins.scm.impl.subversion.SVNRepositoryView.<init>(SVNRepositoryView.java:105)
            at jenkins.scm.impl.subversion.SubversionSCMSource.openSession(SubversionSCMSource.java:338)
            at jenkins.scm.impl.subversion.SubversionSCMSource.retrieve(SubversionSCMSource.java:294)
            at jenkins.scm.api.SCMSource.fetch(SCMSource.java:636)
            at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.lambda$retrieve$0(SCMSourceRetriever.java:89)
            at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrySCMOperation(SCMSourceRetriever.java:104)
            at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:89)
            at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:157)
            at org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:138)
            at org.jenkinsci.plugins.workflow.libs.LibraryDecorator$1.call(LibraryDecorator.java:125)
            at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1065)
            at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
            at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
            at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
            at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
            at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
            at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
            at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
            at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142)
            at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127)
            at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:561)
            at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:522)
            at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:320)
            at hudson.model.ResourceController.execute(ResourceController.java:97)
            at hudson.model.Executor.run(Executor.java:429)
    ...
    2019-07-26 08:01:00.872+0000 [id=148]   INFO    o.j.p.workflow.job.WorkflowRun#finish: Sandbox/ACME.buildDockerImage #1 completed: FAILURE
    2019-07-26 08:01:00.872+0000 [id=146]   INFO    o.j.p.workflow.job.WorkflowRun#finish: Sandbox/ACMD.stage #1 completed: FAILURE
    
    • Re-triggering the failing builds worked

I guess the used org.mapdb v1.0.9 is to blame: there are 3.x releases and quite a few (resolved and unresolved) concurrency issues...

Alternatively, I think, one might also be able to avoid the concurrency problem by locking the cache file creation by oneself in the usage of org.mapdb.DBMaker, that is in jenkins.scm.impl.subversion.SVNRepositoryView.<init>?

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

kuisathaverat@gmail.com (JIRA)

unread,
Jul 26, 2019, 6:43:03 AM7/26/19
to jenkinsc...@googlegroups.com

r.fuereder@xortex.com (JIRA)

unread,
Jul 26, 2019, 7:44:02 AM7/26/19
to jenkinsc...@googlegroups.com
Reinhold Füreder commented on Bug JENKINS-58671
 
Re: Concurrency problems with SVN workflow libs => maybe upgrade MapDB

Also seen:

2019-07-26 11:40:35.928+0000 [id=152]   WARNING j.s.i.s.SVNRepositoryView#<init>: failing to make/load /var/jenkins_home/caches/svn-fa86427e-e420-4d14-a4fd-9ab1bdfcde57.db
java.io.IOException: storage has invalid header
Caused: java.io.IOError
        at org.mapdb.StoreDirect.checkHeaders(StoreDirect.java:254)
        at org.mapdb.StoreWAL.checkHeaders(StoreWAL.java:115)
        at org.mapdb.StoreDirect.<init>(StoreDirect.java:207)
        at org.mapdb.StoreWAL.<init>(StoreWAL.java:74)
        at org.mapdb.DBMaker.extendStoreWAL(DBMaker.java:981)
        at org.mapdb.DBMaker.makeEngine(DBMaker.java:758)
        at org.mapdb.DBMaker.make(DBMaker.java:701)
        at jenkins.scm.impl.subversion.SVNRepositoryView.<init>(SVNRepositoryView.java:105)
        at jenkins.scm.impl.subversion.SubversionSCMSource.openSession(SubversionSCMSource.java:338)
        at jenkins.scm.impl.subversion.SubversionSCMSource.retrieve(SubversionSCMSource.java:294)
        at jenkins.scm.api.SCMSource.fetch(SCMSource.java:636)
        at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.lambda$retrieve$0(SCMSourceRetriever.java:89)
        at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrySCMOperation(SCMSourceRetriever.java:104)
        at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:89)
        at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:157)
        at org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:138)
        at org.jenkinsci.plugins.workflow.libs.LibraryDecorator$1.call(LibraryDecorator.java:125)
        at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1065)
        at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
        at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
        at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
        at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
        at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
        at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
        at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
        at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142)
        at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127)
        at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:561)
        at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:522)
        at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:320)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
Reply all
Reply to author
Forward
0 new messages