Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-34256

Preparing Jenkins For Shutdown Hangs Running Pipelines

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Start a couple long-running pipelines with

      node

      { sleep 100 }

      Queue up a few more jobs. Go to "manage jenkins" and "prepare for shutdown."

      Now pipeline jobs that would finish and unenqueue never finish and have to manually be killed (which does work). Freestyle jobs complete normally. Queued jobs aren't run, so that part of prepare-for-shutdown works.

      Even stranger: upon killing and restarting with Ctrl+C, we get this lovely conundrum:

      Those pipeline builds won't show up in the build queue on the main screen.

      Checks to do:

      • Regression in core?
      • Regression in pipeline?
      • does /safeRestart or /restart trigger it?

        Attachments

          Issue Links

            Activity

            Hide
            hgholami Hamid Gholami added a comment -

            Any update?

            We have same issue on Jenkins.

            Show
            hgholami Hamid Gholami added a comment - Any update? We have same issue on Jenkins.
            Hide
            tknerr Torben Knerr added a comment - - edited

            Same issue here with Jenkins LTS 2.150.2

            I'm seeing this with pipeline durability set to "PERFORMANCE_OPTIMIZED" in the global configuration.

            Sam Van Oort re-reading your comment here https://issues.jenkins-ci.org/browse/JENKINS-34256?focusedCommentId=332080&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-332080 I am wondering why the currently executing pipeline should actually halt by design – wouldn't it be more intuitive if any running pipelines just complete (as it was the case with freestyle jobs earlier)

            Show
            tknerr Torben Knerr added a comment - - edited Same issue here with Jenkins LTS 2.150.2 I'm seeing this with pipeline durability set to "PERFORMANCE_OPTIMIZED" in the global configuration. Sam Van Oort re-reading your comment here https://issues.jenkins-ci.org/browse/JENKINS-34256?focusedCommentId=332080&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-332080 I am wondering why the currently executing pipeline should actually halt by design – wouldn't it be more intuitive if any running pipelines just complete (as it was the case with freestyle jobs earlier)
            Hide
            laszlog Laszlo Gaal added a comment -

            We are also seeing this on LTS 2.150.2 quite regularly.

            Show
            laszlog Laszlo Gaal added a comment - We are also seeing this on LTS 2.150.2 quite regularly.
            Hide
            awkspace awk space added a comment -

            Having the same issue here on Jenkins 2.165 - even with simple 'sh "sleep 60"' test jobs.

            Attempting to work around the issue by checking "Do not allow the pipeline to resume after master restarts" and changing the pipeline to PERFORMANCE_OPTIMIZED makes the pipeline attempt to resume after restart (??) and makes me run into JENKINS-50407 instead.

            Show
            awkspace awk space added a comment - Having the same issue here on Jenkins 2.165 - even with simple 'sh "sleep 60"' test jobs. Attempting to work around the issue by checking "Do not allow the pipeline to resume after master restarts" and changing the pipeline to PERFORMANCE_OPTIMIZED makes the pipeline attempt to resume after restart (??) and makes me run into  JENKINS-50407 instead.
            Hide
            buuren Vladimir K added a comment -

            According to my observations, the bug only affects Jenkins pipelines and happens when a Jenkins is put into shutdown mode when there are some pipelines running on background; those pipelines will not be able to proceed to next pipeline stage(s) and will indefinitely stuck in whatever last stage there was prior shutdown mode.

            This can be reproduced with the following pseudo-pipeline:

            stages {
                stage('build') {
                    steps {
                        sh('make build')
                    }
                }
            
                stage('prepare') {
                    steps {
                        //During this stage, Jenkins is put into shutdown mode
                        sh('make prepare-for-restart')
                    }
                }
            
                stage('deploy') {
                    steps {
                        // Pipeline will not reach this stage
                        sh('make deploy')
                    }
                }
            }
            
            post {
                always {
                    sh('echo Test')
                }
            }

            The following pipeline will never reach neither deploy or post stages

            My guess shutdown prevents any new build threads to be executed and since each stage runs in separate thread (for serialization purposes), pipelines get stuck. This behavior seems to be intended because this allows Jenkins to continue stages after hard restart.

            In my use-case I would like to conduct a safe, controlled Jenkins restart, allowing any existing workloads to finish. 

            Show
            buuren Vladimir K added a comment - According to my observations, the bug only affects Jenkins pipelines and happens when a Jenkins is put into shutdown mode when there are some pipelines running on background; those pipelines will not be able to proceed to next pipeline stage(s) and will indefinitely stuck in whatever last stage there was prior shutdown mode. This can be reproduced with the following pseudo-pipeline: stages { stage( 'build' ) { steps { sh( 'make build' ) } } stage( 'prepare' ) { steps { //During this stage, Jenkins is put into shutdown mode sh( 'make prepare- for -restart' ) } } stage( 'deploy' ) { steps { // Pipeline will not reach this stage sh( 'make deploy' ) } } } post { always { sh( 'echo Test' ) } } The following pipeline will never reach neither deploy  or post stages My guess shutdown prevents any new build threads to be executed and since each stage runs in separate thread (for serialization purposes), pipelines get stuck. This behavior seems to be intended because this allows Jenkins to continue stages after hard restart. In my use-case I would like to conduct a safe, controlled Jenkins restart, allowing any existing workloads to finish. 

              People

              • Assignee:
                Unassigned
                Reporter:
                svanoort Sam Van Oort
              • Votes:
                33 Vote for this issue
                Watchers:
                55 Start watching this issue

                Dates

                • Created:
                  Updated: