Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60039

durable-task v1.33 breaks sh steps in pipeline when running in a Docker container

    Details

    • Similar Issues:

      Description

      Tested on Durable Task Plugin v. 1.29 & v. 1.30 wasn't worked.

      Durable Task Plugin has been "updated to v. 1.33". Still the problem isn't resolved.

       

      PIPELINE:

      pipeline {
          agent {
               docker

      {             label 'buildserver'             image 'busybox'             args '-v /var/lib/jenkins/caches:/jenkins/caches'            args '-v /var/lib/jenkins/workspace:/jenkins/workspace'            args '-u root'                  }

                  }
         stages {
            stage ("Test sh script in container") {
               steps

      {              sh label: 'Echo "Hello World...', script: 'echo "Hello World!"'                            }

                        }
                  }
          }

       

      ERROR:
      [Pipeline] Start of Pipeline

      [Pipeline] nodeRunning on buildserver in c:/jenkins/workspace/<project-name>

      [Pipeline] {

      [Pipeline] stage

      [Pipeline]

      { (Declarative: Checkout SCM) [Pipeline] checkout [Pipeline] }

      [Pipeline] // stage

      [Pipeline] withEnv

      [Pipeline]

      { [Pipeline] isUnix[Pipeline] sh process apparently never started in c:/jenkins/workspace/<project-name>@tmp/durable-73ba6e89 (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer) [Pipeline] }

      [Pipeline] // withEnv

      [Pipeline] }

      [Pipeline] // node

      [Pipeline] End of Pipeline

      ERROR: script returned exit code -2
      Finished: FAILURE

        Attachments

          Activity

          Hide
          aikatz Alex Katz added a comment - - edited

          I'm encountering the same error message on Ubuntu 18.04 except it seems to be breaking a git checkout step (checkout scm) and possibly other sh steps, not in a docker container. Also running durable-task v1.33 but also running lots of other plugins and I haven't tried reproducing the issue on clean system yet.

          Update: Rebuilt Jenkins (with the same plugins and configuration) and now all of the pipelines work as expected, not sure what was causing it to break before.

          Show
          aikatz Alex Katz added a comment - - edited I'm encountering the same error message on Ubuntu 18.04 except it seems to be breaking a git checkout step ( checkout scm ) and possibly other sh steps, not in a docker container. Also running durable-task v1.33 but also running lots of other plugins and I haven't tried reproducing the issue on clean system yet. Update: Rebuilt Jenkins (with the same plugins and configuration) and now all of the pipelines work as expected, not sure what was causing it to break before.
          Hide
          carroll Carroll Chiou added a comment -

          Just noting that the title is a bit misleading. If I read this right, it looks like you have a failed shell step and that it has not been working on 1.29, 1.30, and 1.33. It's not that the upgrade to 1.33 has broken your pipeline correct?

          Is there any way you can run your pipeline again with org.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true and posting the output here? That might give a better idea of what's going on.

          Show
          carroll Carroll Chiou added a comment - Just noting that the title is a bit misleading. If I read this right, it looks like you have a failed shell step and that it has not been working on 1.29, 1.30, and 1.33. It's not that the upgrade to 1.33 has broken your pipeline correct? Is there any way you can run your pipeline again with org.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true and posting the output here? That might give a better idea of what's going on.
          Hide
          timblaktu Tim Black added a comment - - edited

          We're having same problem, but it's not yet clear what change caused the behavior. We are on durable task plugin v1.33, and Jenkins ver. 2.190.3, however we have not updated this recently to my knowledge (and I haven't been able to find plugin update history anywhere - any advice?).

          The signature of the build failures is that the `withDockerContainer` step completes successfully, and the pipeline begins to run the sh command (the first step in the `withDockerContainer` clause) and there's no apparent activity (other than the seemingly normal output in the durable task plugin loggers) for 5 minutes, then the failure message as the OP here. Here is what the durable task loggers output during those 5 minutes:

           

          Mar 05, 2020 7:03:57 AM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep  seems to be online so using /var/lib/jenkins/workspace/TesiraFirmwareNG_develop Mar 05, 2020 7:03:57 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep still running in /var/lib/jenkins/workspace/TesiraFirmwareNG_develop on

          After 5 minutes, I then see:

           

           

          Mar 05, 2020 7:05:12 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
           calling close with nl=true
           Mar 05, 2020 7:05:14 AM FINEST org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
           JENKINS-34021: DurableTaskStep.Execution.listener present in CpsStepContext[101:sh]:OwnerTesiraFirmwareNG/develop/42:TesiraFirmwareNG/develop #42
           Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
           launching task against hudson.remoting.LocalChannel@41a7fb38 using hudson.Launcher$LocalLauncher@722b273b
           Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.BourneShellScript
           launching [nohup, sh, c, ({ while [ -d '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859' -a ! -f '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-result.txt' ]; do touch '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-log.txt'; sleep 3; done } & jsc=durable-97017349f4d4de030acf9aecc2fb49fa; JENKINS_SERVER_COOKIE=$$jsc 'sh' -xe '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/script.sh' > '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-log.txt' 2>&1; echo $$? > '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-result.txt.tmp'; mv '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-result.txt.tmp' '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-result.txt'; wait) >& 2>&- &]
           Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
           launched task
           Mar 05, 2020 7:05:14 AM FINEST org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
           JENKINS-34021: DurableTaskStep.Execution.listener present in CpsStepContext[102:sh]:OwnerTesiraFirmwareNG/develop/42:TesiraFirmwareNG/develop #42
           Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
           launching task against hudson.remoting.LocalChannel@41a7fb38 using hudson.Launcher$LocalLauncher@460edded
           Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.BourneShellScript
           launching [nohup, sh, c, ({ while [ -d '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0' -a ! -f '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-result.txt' ]; do touch '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-log.txt'; sleep 3; done } & jsc=durable-97017349f4d4de030acf9aecc2fb49fa; JENKINS_SERVER_COOKIE=$$jsc 'sh' -xe '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/script.sh' > '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-log.txt' 2>&1; echo $$? > '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-result.txt.tmp'; mv '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-result.txt.tmp' '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-result.txt'; wait) >& 2>&- &]
           Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
           launched task
           Mar 05, 2020 7:05:14 AM FINEST org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
           JENKINS-34021: DurableTaskStep.Execution.listener present in CpsStepContext[103:sh]:OwnerTesiraFirmwareNG/develop/42:TesiraFirmwareNG/develop #42
           Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
           launching task against hudson.remoting.LocalChannel@41a7fb38 using hudson.Launcher$LocalLauncher@6864647
           Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.BourneShellScript
           launching [nohup, sh, c, ({ while [ -d '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f' -a ! -f '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-result.txt' ]; do touch '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-log.txt'; sleep 3; done } & jsc=durable-97017349f4d4de030acf9aecc2fb49fa; JENKINS_SERVER_COOKIE=$$jsc 'sh' -xe '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/script.sh' > '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-log.txt' 2>&1; echo $$? > '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-result.txt.tmp'; mv '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-result.txt.tmp' '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-result.txt'; wait) >& 2>&- &]
           Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
           launched task
           Mar 05, 2020 7:05:14 AM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
           seems to be online so using /var/lib/jenkins/workspace/TesiraFirmwareNG_develop
           Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
           creating filtered stream
           Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.FileMonitoringTask
           remote transcoding charset: null
           Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.FileMonitoringTask
           copied 686 bytes from /var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-log.txt
           Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.BourneShellScript
           found exit code 0 in /var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859
           Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
           calling close with nl=true
           Mar 05, 2020 7:05:14 AM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
           seems to be online so using /var/lib/jenkins/workspace/TesiraFirmwareNG_develop
           Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
           creating filtered stream
           Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.FileMonitoringTask
           remote transcoding charset: null
           Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.FileMonitoringTask
           copied 334 bytes from /var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-log.txt
           Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.BourneShellScript
           found exit code 0 in /var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0
           Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
           calling close with nl=true
           Mar 05, 2020 7:05:14 AM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
           seems to be online so using /var/lib/jenkins/workspace/TesiraFirmwareNG_develop
           Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
           creating filtered stream
           Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.FileMonitoringTask
           remote transcoding charset: null
           Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.FileMonitoringTask
           copied 189 bytes from /var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-log.txt
           Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.BourneShellScript
           found exit code 0 in /var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f
           Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
           calling close with nl=true
          

          Nothing looks suspicious to me here, but I'm unfamiliar with this plugin before last night. Let me know if you can glean anything from this.

          I will look into running with the LAUNCH_DIAGNOSTICS suggestion, but in the meantime, since the changelogs singe 1.29 for this plugin indicate instability, I'm inclined to revert and test.

          Let me know if you have other suggestions.

          Show
          timblaktu Tim Black added a comment - - edited We're having same problem, but it's not yet clear what change caused the behavior. We are on durable task plugin v1.33, and Jenkins ver. 2.190.3 , however we have not updated this recently to my knowledge (and I haven't been able to find plugin update history anywhere - any advice?). The signature of the build failures is that the `withDockerContainer` step completes successfully, and the pipeline begins to run the sh command (the first step in the `withDockerContainer` clause) and there's no apparent activity (other than the seemingly normal output in the durable task plugin loggers) for 5 minutes, then the failure message as the OP here. Here is what the durable task loggers output during those 5 minutes:   Mar 05, 2020 7:03:57 AM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep  seems to be online so using / var /lib/jenkins/workspace/TesiraFirmwareNG_develop Mar 05, 2020 7:03:57 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep still running in / var /lib/jenkins/workspace/TesiraFirmwareNG_develop on After 5 minutes, I then see:     Mar 05, 2020 7:05:12 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep calling close with nl= true Mar 05, 2020 7:05:14 AM FINEST org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep JENKINS-34021: DurableTaskStep.Execution.listener present in CpsStepContext[101:sh]:OwnerTesiraFirmwareNG/develop/42:TesiraFirmwareNG/develop #42 Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep launching task against hudson.remoting.LocalChannel@41a7fb38 using hudson.Launcher$LocalLauncher@722b273b Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.BourneShellScript launching [nohup, sh, c, ({ while [ -d '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859' -a ! -f '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-result.txt' ]; do touch '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-log.txt' ; sleep 3; done } & jsc=durable-97017349f4d4de030acf9aecc2fb49fa; JENKINS_SERVER_COOKIE=$$jsc 'sh' -xe '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/script.sh' > '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-log.txt' 2>&1; echo $$? > '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-result.txt.tmp' ; mv '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-result.txt.tmp' '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-result.txt' ; wait) >& 2>&- &] Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep launched task Mar 05, 2020 7:05:14 AM FINEST org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep JENKINS-34021: DurableTaskStep.Execution.listener present in CpsStepContext[102:sh]:OwnerTesiraFirmwareNG/develop/42:TesiraFirmwareNG/develop #42 Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep launching task against hudson.remoting.LocalChannel@41a7fb38 using hudson.Launcher$LocalLauncher@460edded Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.BourneShellScript launching [nohup, sh, c, ({ while [ -d '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0' -a ! -f '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-result.txt' ]; do touch '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-log.txt' ; sleep 3; done } & jsc=durable-97017349f4d4de030acf9aecc2fb49fa; JENKINS_SERVER_COOKIE=$$jsc 'sh' -xe '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/script.sh' > '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-log.txt' 2>&1; echo $$? > '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-result.txt.tmp' ; mv '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-result.txt.tmp' '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-result.txt' ; wait) >& 2>&- &] Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep launched task Mar 05, 2020 7:05:14 AM FINEST org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep JENKINS-34021: DurableTaskStep.Execution.listener present in CpsStepContext[103:sh]:OwnerTesiraFirmwareNG/develop/42:TesiraFirmwareNG/develop #42 Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep launching task against hudson.remoting.LocalChannel@41a7fb38 using hudson.Launcher$LocalLauncher@6864647 Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.BourneShellScript launching [nohup, sh, c, ({ while [ -d '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f' -a ! -f '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-result.txt' ]; do touch '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-log.txt' ; sleep 3; done } & jsc=durable-97017349f4d4de030acf9aecc2fb49fa; JENKINS_SERVER_COOKIE=$$jsc 'sh' -xe '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/script.sh' > '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-log.txt' 2>&1; echo $$? > '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-result.txt.tmp' ; mv '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-result.txt.tmp' '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-result.txt' ; wait) >& 2>&- &] Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep launched task Mar 05, 2020 7:05:14 AM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep seems to be online so using / var /lib/jenkins/workspace/TesiraFirmwareNG_develop Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep creating filtered stream Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.FileMonitoringTask remote transcoding charset: null Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.FileMonitoringTask copied 686 bytes from / var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-log.txt Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.BourneShellScript found exit code 0 in / var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859 Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep calling close with nl= true Mar 05, 2020 7:05:14 AM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep seems to be online so using / var /lib/jenkins/workspace/TesiraFirmwareNG_develop Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep creating filtered stream Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.FileMonitoringTask remote transcoding charset: null Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.FileMonitoringTask copied 334 bytes from / var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-log.txt Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.BourneShellScript found exit code 0 in / var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0 Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep calling close with nl= true Mar 05, 2020 7:05:14 AM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep seems to be online so using / var /lib/jenkins/workspace/TesiraFirmwareNG_develop Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep creating filtered stream Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.FileMonitoringTask remote transcoding charset: null Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.FileMonitoringTask copied 189 bytes from / var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-log.txt Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.BourneShellScript found exit code 0 in / var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep calling close with nl= true Nothing looks suspicious to me here, but I'm unfamiliar with this plugin before last night. Let me know if you can glean anything from this. I will look into running with the LAUNCH_DIAGNOSTICS suggestion, but in the meantime, since the changelogs singe 1.29 for this plugin indicate instability, I'm inclined to revert and test. Let me know if you have other suggestions.
          Hide
          timblaktu Tim Black added a comment - - edited

          Reverting DT plugin to v1.29, restarting Jenkins, and re building a test job that uses withDockerContainer { sh }, results in an error due to an unmet dependency between Durable Task and Pipeline: Nodes and Processes. So I went back to DT v1.33 since a lot of plugins depend on Nodes and Processes, and I'm not immediately sure how to resolve this dependency heck.

          In the meantime, I have noticed that we are using PERFORMANCE_OPTIMIZED durability level for this job. That seems relevant, but I don't know enough to connect the dots. Also what seems relevant is that this behavior started after a reboot of the Jenkins master while a job downstream of this test job was running on a slave. From what I've read about Durable Task plugin, its purpose is to enable nodes to manage/monitor processes running on other nodes. This smells rotten. Please enlighten me if you can connect any of these dots. 

          UPDATE: I have changed the global durability setting to "MAX_SURVIVABILITY" from "PERFORMANCE_OPTIMIZED" and am still getting the same error.

          Show
          timblaktu Tim Black added a comment - - edited Reverting DT plugin to v1.29, restarting Jenkins, and re building a test job that uses withDockerContainer { sh }, results in an error due to an unmet dependency between Durable Task and Pipeline: Nodes and Processes. So I went back to DT v1.33 since a lot of plugins depend on Nodes and Processes, and I'm not immediately sure how to resolve this dependency heck. In the meantime, I have noticed that we are using PERFORMANCE_OPTIMIZED durability level for this job. That seems relevant, but I don't know enough to connect the dots. Also what seems relevant is that this behavior started after a reboot of the Jenkins master  while a job downstream of this test job was running on a slave. From what I've read about Durable Task plugin, its purpose is to enable nodes to manage/monitor processes running on other nodes. This smells rotten. Please enlighten me if you can connect any of these dots.  UPDATE: I have changed the global durability setting to "MAX_SURVIVABILITY" from "PERFORMANCE_OPTIMIZED" and am still getting the same error.
          Hide
          carroll Carroll Chiou added a comment -

          Tim Black So the differences of 1.29 and 1.33 should be fairly negligible. What happened between all those release was the introduction of a binary wrapper (to replace the shell wrapper) for shell scripts to make them more "durable". However, there were varied compatibility issues (mostly with docker and non-x86 builds) that we made the original shell wrapper the default behavior again. Your logging shows that the original shell wrapper is indeed running. The biggest difference I can see between yours and the OP is that your script appears to be executing (writing output to the jenkins-log.txt) and terminating with 0 as opposed to failure.

          Like I mentioned with OP, if you can run with LAUNCH_DIAGNOSTICS, that might help us get a better idea of what is going on.

          Show
          carroll Carroll Chiou added a comment - Tim Black So the differences of 1.29 and 1.33 should be fairly negligible. What happened between all those release was the introduction of a binary wrapper (to replace the shell wrapper) for shell scripts to make them more "durable". However, there were varied compatibility issues (mostly with docker and non-x86 builds) that we made the original shell wrapper the default behavior again. Your logging shows that the original shell wrapper is indeed running. The biggest difference I can see between yours and the OP is that your script appears to be executing (writing output to the jenkins-log.txt) and terminating with 0 as opposed to failure. Like I mentioned with OP, if you can run with LAUNCH_DIAGNOSTICS, that might help us get a better idea of what is going on.
          Hide
          timblaktu Tim Black added a comment -

          For me, this issue turned out to be an empty environment variable set in Jenkins global properties. This

          Show
          timblaktu Tim Black added a comment - For me, this issue turned out to be an empty environment variable set in Jenkins global properties. This . 

            People

            • Assignee:
              csanchez Carlos Sanchez
              Reporter:
              srinidhisivakumar srinidhi Sivakumar
            • Votes:
              3 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated: