Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38769

Pipeline job hangs when agent loses network connectivity

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Environment:
    • Similar Issues:

      Description

      When running a Pipeline job (see end of this description) while the agent (swarm client v2.0) loses network connectivity (in this case connectivity is lost for a long while), the Pipeline hangs, showing the following in the console log:

      Could not connect to <AGENT NAME> to send interrupt signal to process
      

      After clicking the abort button (/stop), I get the same message printed in the log again and then the "Click here to terminate message":

      Could not connect to <AGENT NAME> to send interrupt signal to process
      Click here to forcibly terminate running steps
      

      After forcing termination, I see the following in the console log:

      Terminating sh
      [Pipeline] }
      [Pipeline] // timeout
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] stage
      [Pipeline] { (Publish Results)
      [Pipeline] step
      09:51:54.766 Required context class hudson.Launcher is missing
      09:51:54.766 Perhaps you forgot to surround the code with a step that provides this, such as: node
      [Pipeline] // stage
      [Pipeline] }
      [Pipeline] // withDockerContainer
      [Pipeline] }
      [Pipeline] // timestamps
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] End of Pipeline
      java.lang.RuntimeException: Failed to serialize org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage$Tag#actions for class org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage$Tag
      	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
      	at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
      	at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
      	at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
      	at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
      	at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82)
      	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
      	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026)
      	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015)
      	at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
      	at hudson.XmlFile.write(XmlFile.java:178)
      	at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage.saveActions(SimpleXStreamFlowNodeStorage.java:109)
      	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.saveActions(CpsFlowExecution.java:887)
      	at org.jenkinsci.plugins.workflow.graph.FlowNode.save(FlowNode.java:315)
      	at org.jenkinsci.plugins.workflow.graph.FlowNode$1.persist(FlowNode.java:302)
      	at org.jenkinsci.plugins.workflow.graph.FlowNode$1.add(FlowNode.java:278)
      	at org.jenkinsci.plugins.workflow.graph.FlowNode$1.add(FlowNode.java:269)
      	at java.util.AbstractList.add(AbstractList.java:108)
      	at hudson.model.Actionable.addAction(Actionable.java:129)
      	at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$FailureAdapter.receive(CpsBodyExecution.java:273)
      	at com.cloudbees.groovy.cps.impl.ContinuationGroup.throwException(ContinuationGroup.java:180)
      	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:54)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
      	at sun.reflect.GeneratedMethodAccessor1031.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      	at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.dispatch(CollectionLiteralBlock.java:55)
      	at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.item(CollectionLiteralBlock.java:45)
      	at sun.reflect.GeneratedMethodAccessor1034.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:103)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
      	at sun.reflect.GeneratedMethodAccessor1031.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      	at performance.Script1.runScript(Script1.groovy:74)
      	at ___cps.transform___(Native Method)
      	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:50)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
      	at sun.reflect.GeneratedMethodAccessor1031.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      	at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
      	at com.cloudbees.groovy.cps.Next.step(Next.java:58)
      	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:154)
      	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:163)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:324)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:78)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:236)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:224)
      	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:63)
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
      	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:791)
      	Suppressed: java.lang.NullPointerException: while trying to invoke the method hudson.Launcher.getListener() of a null object loaded from field org.jenkinsci.plugins.docker.workflow.client.DockerClient.launcher of an object loaded from local variable 'this'
      		at org.jenkinsci.plugins.docker.workflow.client.DockerClient.launch(DockerClient.java:251)
      		at org.jenkinsci.plugins.docker.workflow.client.DockerClient.launch(DockerClient.java:247)
      		at org.jenkinsci.plugins.docker.workflow.client.DockerClient.launch(DockerClient.java:244)
      		at org.jenkinsci.plugins.docker.workflow.client.DockerClient.stop(DockerClient.java:139)
      		at org.jenkinsci.plugins.docker.workflow.WithContainerStep.destroy(WithContainerStep.java:104)
      		at org.jenkinsci.plugins.docker.workflow.WithContainerStep.access$400(WithContainerStep.java:71)
      		at org.jenkinsci.plugins.docker.workflow.WithContainerStep$Callback.finished(WithContainerStep.java:281)
      		at org.jenkinsci.plugins.workflow.steps.BodyExecutionCallback$TailCall.onFailure(BodyExecutionCallback.java:124)
      		at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$FailureAdapter.receive(CpsBodyExecution.java:278)
      		at com.cloudbees.groovy.cps.impl.ThrowBlock$1.receive(ThrowBlock.java:68)
      		... 19 more
      Caused by: java.lang.RuntimeException: Failed to serialize org.jenkinsci.plugins.workflow.actions.ErrorAction#error for class org.jenkinsci.plugins.workflow.actions.ErrorAction
      	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
      	at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
      	at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
      	at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
      	at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88)
      	at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:64)
      	at com.thoughtworks.xstream.converters.collections.ArrayConverter.marshal(ArrayConverter.java:45)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
      	at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265)
      	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252)
      	at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
      	at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
      	at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
      	at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
      	at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82)
      	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
      	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026)
      	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015)
      	at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
      	at hudson.XmlFile.write(XmlFile.java:178)
      	at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage.saveActions(SimpleXStreamFlowNodeStorage.java:109)
      	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.saveActions(CpsFlowExecution.java:887)
      	at org.jenkinsci.plugins.workflow.graph.FlowNode.save(FlowNode.java:315)
      	at org.jenkinsci.plugins.workflow.graph.FlowNode$1.persist(FlowNode.java:302)
      	at org.jenkinsci.plugins.workflow.graph.FlowNode$1.add(FlowNode.java:278)
      	at org.jenkinsci.plugins.workflow.graph.FlowNode$1.add(FlowNode.java:269)
      	at java.util.AbstractList.add(AbstractList.java:108)
      	at hudson.model.Actionable.addAction(Actionable.java:129)
      	at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$FailureAdapter.receive(CpsBodyExecution.java:273)
      	at com.cloudbees.groovy.cps.impl.ContinuationGroup.throwException(ContinuationGroup.java:180)
      	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:54)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
      	at sun.reflect.GeneratedMethodAccessor1031.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      	at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.dispatch(CollectionLiteralBlock.java:55)
      	at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.item(CollectionLiteralBlock.java:45)
      	at sun.reflect.GeneratedMethodAccessor1034.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:103)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
      	at sun.reflect.GeneratedMethodAccessor1031.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      	... 26 more
      Caused by: java.lang.RuntimeException: Failed to serialize java.lang.Throwable#cause for class java.lang.IllegalArgumentException
      	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
      	at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
      	at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
      	at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
      	at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
      	at com.thoughtworks.xstream.converters.extended.ThrowableConverter.marshal(ThrowableConverter.java:62)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
      	at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265)
      	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252)
      	... 83 more
      Caused by: java.lang.UnsupportedOperationException: Refusing to marshal org.codehaus.groovy.runtime.InvokerInvocationException for security reasons
      	at hudson.util.XStream2$BlacklistedTypesConverter.marshal(XStream2.java:442)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
      	at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265)
      	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252)
      	... 93 more
      Finished: ABORTED
      

      Pipeline is defined like this:
      Main Script:

      def pipelineScript
      node('my_node') {
          git changelog: false, credentialsId: '...', poll: false, url: 'ssh://...'
          pipelineScript = load '.....groovy'
      }
      
      pipelineScript.runScript()
      

      Script1:

      def runScript() {
          node("my_other_node") {
      
              timestamps {
      
                  stage('S1') {
                      seleniumHost = sh script: 'hostname --all-fqdns', returnStdout: true
                      seleniumHost = seleniumHost.stripIndent().trim()
      
                      env.PATH = '/usr/lib/jvm/jdk1.8.0_60//bin:$PATH:/usr/local/bin/'
                  }
      
                  stage('Checkout Git') {
                      checkout(
                              [
                                      $class           : 'GitSCM', branches: [[name: "${GERRIT_BRANCH}"]],
                                      userRemoteConfigs: [[credentialsId: '...', url: "ssh://...", refspec: "${REFSPEC}"]],
                              ])
                  }
                  stage('S3') {
                      sh """Some script""".stripIndent().trim()
      
                      sh """Some other script""".stripIndent().trim()
                  }
      
                  docker.image('...').inside('') {
      
                      try {
                          stage('S4') {
                              sh """
                              ...
                              npm install
                        """.stripIndent().trim()
                          }
      
                          stage('Run Tests") {
                              timeout(20) {
                                  sh """grunt run_tests""".stripIndent().trim()
                              }
                          }
                      } finally {
                          stage('Publish Results') {
                              archiveArtifacts artifacts: "**/${resultsFileName}"
                              junit "**/${resultsFileName}"
                          }
      
                      }
                  }
      
              }
          }
      }
      
      return this
      

      Thanks!

        Attachments

          Issue Links

            Activity

            yotamshapira Yotam Shapira created issue -
            jglick Jesse Glick made changes -
            Field Original Value New Value
            Component/s workflow-durable-task-step-plugin [ 21715 ]
            Component/s pipeline [ 21692 ]
            Component/s workflow-support-plugin [ 21719 ]
            jglick Jesse Glick made changes -
            Link This issue duplicates JENKINS-34488 [ JENKINS-34488 ]
            jglick Jesse Glick made changes -
            Priority Critical [ 2 ] Major [ 3 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-40613 [ JENKINS-40613 ]
            jglick Jesse Glick made changes -
            Epic Link JENKINS-35399 [ 171192 ]
            jglick Jesse Glick made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-37730 [ JENKINS-37730 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "PR 23 (Web Link)" [ 15168 ]
            jglick Jesse Glick made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            recampbell Ryan Campbell made changes -
            Labels pipeline-hangs robustness
            jglick Jesse Glick made changes -
            Status In Review [ 10005 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-32228 [ JENKINS-32228 ]

              People

              • Assignee:
                Unassigned
                Reporter:
                yotamshapira Yotam Shapira
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: