Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-58671

Concurrency problems with SVN workflow libs => maybe upgrade MapDB

    Details

    • Similar Issues:

      Description

      Initial Docker based Jenkins startup based on JCasC and job-dsl and pipeline jobs auto-scheduled (via job-dsl 'queue' command) right after their creation that all use a shared Jenkins pipeline library in SVN.

      • This did not hurt and led to successful pipeline job executions
        2019-07-26 08:16:12.952+0000 [id=150]   WARNING j.s.i.s.SVNRepositoryView#<init>: failing to make/load /var/jenkins_home/caches/svn-fa86427e-e420-4d14-a4fd-9ab1bdfcde57.db
        java.lang.AssertionError: unknown trans log instruction '-58' at log offset: 176
                at org.mapdb.StoreWAL.replayLogFile(StoreWAL.java:827)
                at org.mapdb.StoreWAL.commit(StoreWAL.java:606)
                at org.mapdb.EngineWrapper.commit(EngineWrapper.java:94)
                at org.mapdb.DBMaker.makeEngine(DBMaker.java:838)
                at org.mapdb.DBMaker.make(DBMaker.java:701)
                at jenkins.scm.impl.subversion.SVNRepositoryView.<init>(SVNRepositoryView.java:105)
                at jenkins.scm.impl.subversion.SubversionSCMSource.openSession(SubversionSCMSource.java:338)
                at jenkins.scm.impl.subversion.SubversionSCMSource.retrieve(SubversionSCMSource.java:294)
                at jenkins.scm.api.SCMSource.fetch(SCMSource.java:636)
                at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.lambda$retrieve$0(SCMSourceRetriever.java:89)
                at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrySCMOperation(SCMSourceRetriever.java:104)
                at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:89)
                at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:157)
                at org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:138)
                at org.jenkinsci.plugins.workflow.libs.LibraryDecorator$1.call(LibraryDecorator.java:125)
                at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1065)
                at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
                at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
                at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
                at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
                at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
                at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
                at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
                at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142)
                at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127)
                at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:561)
                at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:522)
                at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:320)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
        
      • This, by contrast, did hurt and led to failing builds:
        2019-07-26 08:01:00.838+0000 [id=150]   WARNING j.s.i.s.SVNRepositoryView#<init>: failing to make/load /var/jenkins_home/caches/svn-fa86427e-e420-4d14-a4fd-9ab1bdfcde57.db
        java.io.EOFException
                at org.mapdb.Volume$FileChannelVol.readFully(Volume.java:947)
                at org.mapdb.Volume$FileChannelVol.getByte(Volume.java:997)
        Caused: java.io.IOError
                at org.mapdb.Volume$FileChannelVol.getByte(Volume.java:1000)
                at org.mapdb.StoreWAL.replayLogFile(StoreWAL.java:797)
                at org.mapdb.StoreWAL.commit(StoreWAL.java:606)
                at org.mapdb.EngineWrapper.commit(EngineWrapper.java:94)
                at org.mapdb.BTreeMap.preinitCatalog(BTreeMap.java:176)
                at org.mapdb.DB.reinit(DB.java:88)
                at org.mapdb.DB.<init>(DB.java:83)
                at org.mapdb.DBMaker.make(DBMaker.java:704)
                at jenkins.scm.impl.subversion.SVNRepositoryView.<init>(SVNRepositoryView.java:105)
                at jenkins.scm.impl.subversion.SubversionSCMSource.openSession(SubversionSCMSource.java:338)
                at jenkins.scm.impl.subversion.SubversionSCMSource.retrieve(SubversionSCMSource.java:294)
                at jenkins.scm.api.SCMSource.fetch(SCMSource.java:636)
                at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.lambda$retrieve$0(SCMSourceRetriever.java:89)
                at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrySCMOperation(SCMSourceRetriever.java:104)
                at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:89)
                at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:157)
                at org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:138)
                at org.jenkinsci.plugins.workflow.libs.LibraryDecorator$1.call(LibraryDecorator.java:125)
                at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1065)
                at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
                at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
                at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
                at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
                at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
                at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
                at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
                at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142)
                at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127)
                at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:561)
                at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:522)
                at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:320)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
        ...
        2019-07-26 08:01:00.872+0000 [id=148]   INFO    o.j.p.workflow.job.WorkflowRun#finish: Sandbox/ACME.buildDockerImage #1 completed: FAILURE
        2019-07-26 08:01:00.872+0000 [id=146]   INFO    o.j.p.workflow.job.WorkflowRun#finish: Sandbox/ACMD.stage #1 completed: FAILURE
        
        • Re-triggering the failing builds worked

      I guess the used org.mapdb v1.0.9 is to blame: there are 3.x releases and quite a few (resolved and unresolved) concurrency issues...

      Alternatively, I think, one might also be able to avoid the concurrency problem by locking the cache file creation by oneself in the usage of org.mapdb.DBMaker, that is in jenkins.scm.impl.subversion.SVNRepositoryView.<init>?

        Attachments

          Activity

          Hide
          reinholdfuereder Reinhold Füreder added a comment -

          Also seen:

          2019-07-26 11:40:35.928+0000 [id=152]   WARNING j.s.i.s.SVNRepositoryView#<init>: failing to make/load /var/jenkins_home/caches/svn-fa86427e-e420-4d14-a4fd-9ab1bdfcde57.db
          java.io.IOException: storage has invalid header
          Caused: java.io.IOError
                  at org.mapdb.StoreDirect.checkHeaders(StoreDirect.java:254)
                  at org.mapdb.StoreWAL.checkHeaders(StoreWAL.java:115)
                  at org.mapdb.StoreDirect.<init>(StoreDirect.java:207)
                  at org.mapdb.StoreWAL.<init>(StoreWAL.java:74)
                  at org.mapdb.DBMaker.extendStoreWAL(DBMaker.java:981)
                  at org.mapdb.DBMaker.makeEngine(DBMaker.java:758)
                  at org.mapdb.DBMaker.make(DBMaker.java:701)
                  at jenkins.scm.impl.subversion.SVNRepositoryView.<init>(SVNRepositoryView.java:105)
                  at jenkins.scm.impl.subversion.SubversionSCMSource.openSession(SubversionSCMSource.java:338)
                  at jenkins.scm.impl.subversion.SubversionSCMSource.retrieve(SubversionSCMSource.java:294)
                  at jenkins.scm.api.SCMSource.fetch(SCMSource.java:636)
                  at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.lambda$retrieve$0(SCMSourceRetriever.java:89)
                  at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrySCMOperation(SCMSourceRetriever.java:104)
                  at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:89)
                  at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:157)
                  at org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:138)
                  at org.jenkinsci.plugins.workflow.libs.LibraryDecorator$1.call(LibraryDecorator.java:125)
                  at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1065)
                  at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
                  at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
                  at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
                  at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
                  at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
                  at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
                  at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
                  at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142)
                  at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127)
                  at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:561)
                  at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:522)
                  at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:320)
                  at hudson.model.ResourceController.execute(ResourceController.java:97)
                  at hudson.model.Executor.run(Executor.java:429)
          
          Show
          reinholdfuereder Reinhold Füreder added a comment - Also seen: 2019-07-26 11:40:35.928+0000 [id=152] WARNING j.s.i.s.SVNRepositoryView#<init>: failing to make/load / var /jenkins_home/caches/svn-fa86427e-e420-4d14-a4fd-9ab1bdfcde57.db java.io.IOException: storage has invalid header Caused: java.io.IOError at org.mapdb.StoreDirect.checkHeaders(StoreDirect.java:254) at org.mapdb.StoreWAL.checkHeaders(StoreWAL.java:115) at org.mapdb.StoreDirect.<init>(StoreDirect.java:207) at org.mapdb.StoreWAL.<init>(StoreWAL.java:74) at org.mapdb.DBMaker.extendStoreWAL(DBMaker.java:981) at org.mapdb.DBMaker.makeEngine(DBMaker.java:758) at org.mapdb.DBMaker.make(DBMaker.java:701) at jenkins.scm.impl.subversion.SVNRepositoryView.<init>(SVNRepositoryView.java:105) at jenkins.scm.impl.subversion.SubversionSCMSource.openSession(SubversionSCMSource.java:338) at jenkins.scm.impl.subversion.SubversionSCMSource.retrieve(SubversionSCMSource.java:294) at jenkins.scm.api.SCMSource.fetch(SCMSource.java:636) at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.lambda$retrieve$0(SCMSourceRetriever.java:89) at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrySCMOperation(SCMSourceRetriever.java:104) at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:89) at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:157) at org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:138) at org.jenkinsci.plugins.workflow.libs.LibraryDecorator$1.call(LibraryDecorator.java:125) at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1065) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603) at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688) at groovy.lang.GroovyShell.parse(GroovyShell.java:700) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:561) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:522) at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:320) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429)

            People

            • Assignee:
              Unassigned
              Reporter:
              reinholdfuereder Reinhold Füreder
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: