Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-34313

SAXParseException from parallel SCM checkout

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Labels:
      None
    • Environment:
      Jenkins ver. 1.651.1
      Pipeline plug-ins: ver. 1.15
      Pipeline: SCM Step plug-in: ver. 1.15
      SCM API Plugin: ver 1.1
      Subversion Plug-in: ver. 2.5.6
    • Similar Issues:
    • Released As:
      workflow-scm-step 2.8

      Description

      In out pipeline we at first build multiple upstream Maven projects and then build the main project. Upstream projects are build in parallel:

      parallel(
          projA: {
              projects.buildDependency('projA')
          },
          projB: {
              projects.buildDependency('projB')
          }
      ...
      )
      

      projects.buildDependency() just checks-out each project from SVN and starts Maven build. The problem is that we often (in about 20% of builds) have errors caused by concurrent / parallel SVN check-outs. Sometimes it is ConcurrentModificationException and sometimes "SAXParseException: changelog11.xml; lineNumber: 1; columnNumber: 1; Premature end of file". See stacktraces below.

      As a temporary workaround we added

      sleep time: random.nextInt(1000), unit: 'MILLISECONDS'
      

      just before the checkout step to introduce time shifts between check-outs. We used random numbers because we were not sure if some local or env counter variable would be thread-safe in case of parallel steps.

      Now error happens only if time difference between check-outs is narrow, e.g. 5 milliseconds if 2 generated random numbers are close.

      Would it be possible to make Pipeline plug-in more resilient in case of parallel check-outs?

      Caused by: org.jenkinsci.plugins.workflow.cps.steps.ParallelStepException: Parallel step projA failed
      	... 26 more
      Caused by: hudson.util.IOException2: Failed to parse /var/lib/jenkins/jobs/ha-pipeline/builds/426/changelog11.xml
      	at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:80)
      	at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:43)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:638)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1200(WorkflowRun.java:111)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:812)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:123)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73)
      	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:49)
      	at hudson.security.ACL.impersonate(ACL.java:213)
      	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:47)
      	... 5 more
      Caused by: org.xml.sax.SAXParseException; systemId: file:/var/lib/jenkins/jobs/ha-pipeline/builds/426/changelog11.xml; lineNumber: 1; columnNumber: 1; Premature end of file.
      	at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239)
      	at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:642)
      	at org.apache.commons.digester.Digester.parse(Digester.java:1871)
      	at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76)
      	... 15 more
      
      org.jenkinsci.plugins.workflow.cps.steps.ParallelStepException: Parallel step projB failed
      	at org.jenkinsci.plugins.workflow.cps.steps.ParallelStep$ResultHandler$Callback.checkAllDone(ParallelStep.java:153)
      	at org.jenkinsci.plugins.workflow.cps.steps.ParallelStep$ResultHandler$Callback.onSuccess(ParallelStep.java:110)
      	at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$SuccessAdapter.receive(CpsBodyExecution.java:311)
      	at com.cloudbees.groovy.cps.impl.LocalVariableBlock$LocalVariable.get(LocalVariableBlock.java:33)
      	at com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30)
      	at com.cloudbees.groovy.cps.impl.LocalVariableBlock.evalLValue(LocalVariableBlock.java:22)
      	at com.cloudbees.groovy.cps.LValueBlock$BlockImpl.eval(LValueBlock.java:55)
      	at com.cloudbees.groovy.cps.LValueBlock.eval(LValueBlock.java:16)
      	at com.cloudbees.groovy.cps.Next.step(Next.java:58)
      	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:154)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30)
      	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:106)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30)
      	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:164)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:277)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$000(CpsThreadGroup.java:77)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:186)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:184)
      	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:47)
      	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      Caused by: java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class org.jenkinsci.plugins.workflow.job.WorkflowRun
      	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:215)
      	at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:183)
      	at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
      	at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:168)
      	at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:109)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
      	at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82)
      	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
      	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026)
      	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015)
      	at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
      	at hudson.XmlFile.write(XmlFile.java:178)
      	at hudson.model.Run.save(Run.java:1896)
      	at hudson.util.PersistedList.onModified(PersistedList.java:173)
      	at hudson.util.PersistedList._onModified(PersistedList.java:181)
      	at hudson.util.PersistedList.add(PersistedList.java:72)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:651)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1200(WorkflowRun.java:111)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:812)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:123)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73)
      	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:49)
      	at hudson.security.ACL.impersonate(ACL.java:213)
      	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:47)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.RuntimeException: Failed to serialize org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState#revisionStates for class org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState
      	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:215)
      	at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:183)
      	at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
      	at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:168)
      	at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:109)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88)
      	at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:64)
      	at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:74)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
      	at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:224)
      	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:211)
      	... 31 more
      Caused by: java.util.ConcurrentModificationException
      	at java.util.HashMap$HashIterator.nextEntry(HashMap.java:922)
      	at java.util.HashMap$EntryIterator.next(HashMap.java:962)
      	at java.util.HashMap$EntryIterator.next(HashMap.java:960)
      	at com.thoughtworks.xstream.converters.collections.MapConverter.marshal(MapConverter.java:75)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
      	at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:224)
      	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:211)
      	... 46 more
      

        Attachments

          Issue Links

            Activity

            Hide
            benjboyer Benjamin Boyer added a comment - - edited

            Hi Devin Nusbaum,

            I'm having the same issue for. I'll use workflow-scm-step-2.8-rc326.7286a182d923.hpi to confirm if it's fixing the "ConcurrentModificationException". I'll test it against 100+ parallelized repositories (in groups of 12 concurrent extractions)

            Thanks in advance

            Show
            benjboyer Benjamin Boyer added a comment - - edited Hi Devin Nusbaum , I'm having the same issue for. I'll use workflow-scm-step-2.8-rc326.7286a182d923.hpi to confirm if it's fixing the "ConcurrentModificationException". I'll test it against 100+ parallelized repositories (in groups of 12 concurrent extractions) Thanks in advance
            Hide
            skojenkins Andre Morales added a comment -

            I was having the same issue with a pipeline of 23 parallel  checkouts. It was failing randomly. 

            I have installed plugin workflow-scm-step-2.8-rc326.7286a182d923.hpi and everything goes well now.  Since installation, all 23 agents running in parallel are checking out correctly.

             

            Show
            skojenkins Andre Morales added a comment - I was having the same issue with a pipeline of 23 parallel  checkouts. It was failing randomly.  I have installed plugin  workflow-scm-step-2.8-rc326.7286a182d923.hpi and everything goes well now.  Since installation, all 23 agents running in parallel are checking out correctly.  
            Hide
            thefifo Wolfgang Issovits added a comment -

            Is there a chance to get a new release of the plugin with this fix included?
            That would be great as this issue is really problematic for larger parallel pipelines.

            Show
            thefifo Wolfgang Issovits added a comment - Is there a chance to get a new release of the plugin with this fix included? That would be great as this issue is really problematic for larger parallel pipelines.
            Hide
            twerner Torsten Werner added a comment -

            Please, release the fix. The issue is really a problem for us.

            Show
            twerner Torsten Werner added a comment - Please, release the fix. The issue is really a problem for us.
            Hide
            dnusbaum Devin Nusbaum added a comment -

            A fix for this issue was just released in Pipeline SCM Step Plugin version 2.8.

            Show
            dnusbaum Devin Nusbaum added a comment - A fix for this issue was just released in Pipeline SCM Step Plugin version 2.8.

              People

              • Assignee:
                dnusbaum Devin Nusbaum
                Reporter:
                gitt Slawa Giterman
              • Votes:
                18 Vote for this issue
                Watchers:
                21 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: