Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56343

Intermittent java.io.NotSerializableException when running scripts in parallel

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: pipeline
    • Labels:
      None
    • Environment:
    • Similar Issues:

      Description

      I receive an intermittent java.io.NotSerializableException when running a script in parallel

       

      an exception which occurred:
      	in field com.cloudbees.groovy.cps.impl.BlockScopeEnv.locals
      	in object com.cloudbees.groovy.cps.impl.BlockScopeEnv@53ce7f91
      ...
      Caused: java.io.NotSerializableException: java.util.regex.Matcher
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:926)
      ...

       

      The stack trace completely masks the location of the offending code. The offending code is when I'm storing the result of a matcher in a local variable, the main issue is that it was very difficult to locate and that it occurs inconsistently.

      I've reduced the steps required to reproduce to the following script:

       

      import com.cloudbees.groovy.cps.NonCPS
      
      //Fails when it enters this method, only if method is annotated NonCPS
      @NonCPS
      def getMessage(def auth) {
          return '[abc][def]asdasd'
      }
      
      def getTags() {
      //Below line is the offending code, it seems to break CPS rules, but it only fails intermittently
          def tagMatches = getMessage(credentials) =~ /\[(.*?)\]/
          def tagsList = []
          for (def tagMatch in tagMatches) {
              tagsList << (tagMatch[1])?.toLowerCase()
          }
          tagsList
      }
      
      //This doesn't seem as though it would be relevant, but if I remove it when reproducing the failure doesn't seem to appear, perhaps it causes some necessary delay that causes serialisation in the parallel run?
      def getCredentials() {
          withCredentials([
                  [$class: 'StringBinding', credentialsId: 'SOME_CREDENTIALS', variable: 'TOKEN']
          ])
                  {
                      "any"
                  }
      }
      
      def script = {
          stage(it) {
              println tags
          }
      }
      
      def scripts = [:]
      
      for (int i = 0; i < 5; i++) {
          def runName = "run ${i}"
          scripts."run ${i}" = { script(runName) }
      }
      
      //Running the scripts sequentially does not cause issue
       for ( def _script:scripts.values()){
           _script()
       }
      //Running the scripts in parallel causes the issue
      parallel scripts
      

       

       

        Attachments

          Activity

          Hide
          aedwards Alun Edwards added a comment -

          Appears to be similar issue as https://issues.jenkins-ci.org/browse/JENKINS-46597

          Although JENKINS-46597 was resolved because they fixed their underlying issue (use of a tree map), e.g. and for me I could resolve this by storing the result of the matcher in a method not marked as non-cps, for me the outstanding issues are that

          1. The issue happens inconsistently, surely the aforementioned code should always be a violation or should always pass
          2. The issue is very hard to locate. The stack trace gives no indication as to where the cause is. In my case it was a regex matcher, which is used in many places in a pipeline library we have. Even adding logging statements on every line it was still difficult, as where the code is actually triggered is (when it enters the @NonCPS method) is not where the problematic variable is.
          Show
          aedwards Alun Edwards added a comment - Appears to be similar issue as https://issues.jenkins-ci.org/browse/JENKINS-46597 Although JENKINS-46597 was resolved because they fixed their underlying issue (use of a tree map), e.g. and for me I could resolve this by storing the result of the matcher in a method not marked as non-cps, for me the outstanding issues are that The issue happens inconsistently, surely the aforementioned code should always be a violation or should always pass The issue is very hard to locate. The stack trace gives no indication as to where the cause is. In my case it was a regex matcher, which is used in many places in a pipeline library we have. Even adding logging statements on every line it was still difficult, as where the code is actually triggered is (when it enters the @NonCPS method) is not where the problematic variable is.
          Hide
          marcellodesales Marcello de Sales added a comment -
              def releasePrs = prMergesMessages.collectEntries{ prShaMessage ->
                  def separate = prShaMessage.split(":")
                  [separate[0], separate[1]]
                }.collect{ entry ->
                  def matcher = entry.value =~ /(?<prNumber>#[0-9]+)/
                  def pr = matcher.find() ? matcher.group("prNumber") : "none"
                  matcher = null
                  [commitId: entry.key, prNumber: pr]
                }

           

          We've been getting this on the same problem around the Matcher... I got the problem to be resolved for a while by setting the matcher to null... However, today we started experiencing the intermittent behavior without code changes...  I'm deciding to add a try/catch in this block with the hopes to not see this error again, but it's definitely difficult to spot the bug the first time due to the horrible logs... (See below)

           

          The code is in a shared library already and it breaks all of our different DSL-based builds...  

           

          
          n exception which occurred:
          	in field com.cloudbees.groovy.cps.impl.BlockScopeEnv.locals
          	in object com.cloudbees.groovy.cps.impl.BlockScopeEnv@63468bc9
          	in field com.cloudbees.groovy.cps.impl.ProxyEnv.parent
          	in object com.cloudbees.groovy.cps.impl.LoopBlockScopeEnv@16c46b7c
          	in field com.cloudbees.groovy.cps.impl.ProxyEnv.parent
          	in object com.cloudbees.groovy.cps.impl.BlockScopeEnv@63fd3e95
          	in field com.cloudbees.groovy.cps.impl.CallEnv.caller
          	in object com.cloudbees.groovy.cps.impl.FunctionCallEnv@47149d74
          	in field com.cloudbees.groovy.cps.Continuable.e
          	in object org.jenkinsci.plugins.workflow.cps.SandboxContinuable@24b7f5ac
          	in field org.jenkinsci.plugins.workflow.cps.CpsThread.program
          	in object org.jenkinsci.plugins.workflow.cps.CpsThread@3b872106
          	in field org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.threads
          	in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@606f1e48
          	in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@606f1e48
          Also:   org.jenkinsci.plugins.workflow.steps.FlowInterruptedException
          		at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution.cancel(CpsBodyExecution.java:253)
          		at org.jenkinsci.plugins.workflow.steps.BodyExecution.cancel(BodyExecution.java:76)
          		at org.jenkinsci.plugins.workflow.cps.steps.ParallelStepExecution.stop(ParallelStepExecution.java:67)
          		at org.jenkinsci.plugins.workflow.cps.CpsThread.stop(CpsThread.java:296)
          		at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$6.onSuccess(CpsFlowExecution.java:1151)
          		at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$6.onSuccess(CpsFlowExecution.java:1140)
          Caused: java.io.NotSerializableException: java.util.regex.Matcher
          	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:926)
          	at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
          	at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
          	at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50)
          	at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179)
          	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
          	at java.util.HashMap.internalWriteEntries(HashMap.java:1793)
          	at java.util.HashMap.writeObject(HashMap.java:1363)
          	at sun.reflect.GeneratedMethodAccessor312.invoke(Unknown Source)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:498)
          ...
          ...
          ...

           

           

           

          Show
          marcellodesales Marcello de Sales added a comment - def releasePrs = prMergesMessages.collectEntries{ prShaMessage -> def separate = prShaMessage.split( ":" ) [separate[0], separate[1]] }.collect{ entry -> def matcher = entry.value =~ /(?<prNumber>#[0-9]+)/ def pr = matcher.find() ? matcher.group( "prNumber" ) : "none" matcher = null [commitId: entry.key, prNumber: pr] }   We've been getting this on the same problem around the Matcher... I got the problem to be resolved for a while by setting the matcher to null... However, today we started experiencing the intermittent behavior without code changes...  I'm deciding to add a try/catch in this block with the hopes to not see this error again, but it's definitely difficult to spot the bug the first time due to the horrible logs... (See below)   The code is in a shared library already and it breaks all of our different DSL-based builds...     n exception which occurred: in field com.cloudbees.groovy.cps.impl.BlockScopeEnv.locals in object com.cloudbees.groovy.cps.impl.BlockScopeEnv@63468bc9 in field com.cloudbees.groovy.cps.impl.ProxyEnv.parent in object com.cloudbees.groovy.cps.impl.LoopBlockScopeEnv@16c46b7c in field com.cloudbees.groovy.cps.impl.ProxyEnv.parent in object com.cloudbees.groovy.cps.impl.BlockScopeEnv@63fd3e95 in field com.cloudbees.groovy.cps.impl.CallEnv.caller in object com.cloudbees.groovy.cps.impl.FunctionCallEnv@47149d74 in field com.cloudbees.groovy.cps.Continuable.e in object org.jenkinsci.plugins.workflow.cps.SandboxContinuable@24b7f5ac in field org.jenkinsci.plugins.workflow.cps.CpsThread.program in object org.jenkinsci.plugins.workflow.cps.CpsThread@3b872106 in field org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.threads in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@606f1e48 in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@606f1e48 Also: org.jenkinsci.plugins.workflow.steps.FlowInterruptedException at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution.cancel(CpsBodyExecution.java:253) at org.jenkinsci.plugins.workflow.steps.BodyExecution.cancel(BodyExecution.java:76) at org.jenkinsci.plugins.workflow.cps.steps.ParallelStepExecution.stop(ParallelStepExecution.java:67) at org.jenkinsci.plugins.workflow.cps.CpsThread.stop(CpsThread.java:296) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$6.onSuccess(CpsFlowExecution.java:1151) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$6.onSuccess(CpsFlowExecution.java:1140) Caused: java.io.NotSerializableException: java.util.regex.Matcher at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:926) at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65) at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56) at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50) at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344) at java.util.HashMap.internalWriteEntries(HashMap.java:1793) at java.util.HashMap.writeObject(HashMap.java:1363) at sun.reflect.GeneratedMethodAccessor312.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) ... ... ...      

            People

            • Assignee:
              Unassigned
              Reporter:
              aedwards Alun Edwards
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: