Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38316

Entering, then cancelling, quiet mode causes builds to hang

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Duplicate
    • Component/s: workflow-cps-plugin
    • Labels:
      None
    • Environment:
      jenkins:latest docker image
    • Similar Issues:

      Description

      This problem can be reproduced with a simple pipeline job that does nothing but echo and sleep in a loop.

      1) Start a run of the pipeline job, the job produces output in the Console Output as I expect
      2) Put Jenkins in "Preparing for Shutdown" mode. The jobs stop producing output. That's somewhat surprising, as runs that were started before entering Shutdown Mode are usually allowed to complete.
      3) Cancel "Shutdown Mode." The job is still "stuck" and doesn't produce any output. The job also cannot be canceled except by a "hard kill," i.e. going through the "Forcibly terminate running steps" and "Forcibly kill entire build" links

      The behavior in #2 is surprising, but not really a problem for me. However, the fact that entering and leaving shutdown mode completely hoses running jobs is really bad. This is compounded by an init.groovy script we use that places Jenkins in shutdown mode at startup.

        Attachments

          Issue Links

            Activity

            Hide
            h0riz0n Brent Goodacre added a comment - - edited

            I've noticed this problem as well, it really makes "Prepare for shutdown" quite useless for us because it's tedious to have to hard kill our currently running jobs. The whole idea of "Prepare for shutdown" is to allow us to finish the currently running jobs, not freeze them where they are.

            It's worth calling out that this has only happened in a recent upgrade of either Jenkins or one of the Pipeline plugins (Sorry it's vague but I'd estimate a change in the last 1-2 months)

            Here is a workflow script to repro:

            1. Create and start a job that uses the following code:

            node( "testnode" )
            {
                for( int i = 0; i < 10; i++ )
                {
                    println "Sleep: ${i+1}"
                    sleep(5)
                }
            }
            

            2. Activate "Prepare for Shutdown" mode while the job is running.

            Expected Result: Job continues to run as it's already midway through execution.
            Actual Result: Job freezes

            Steven Walter this might not help you, but if you cancel "Prepare for Shutdown" and then on your job click "Pause/resume" to toggle the job into Paused, then Resumed states, the job will continue. Eg.

            Sleep: 3
            [Pipeline] sleep
            Sleeping for 5 sec
            Pausing
            Resuming
            [Pipeline] echo
            Sleep: 4
            [Pipeline] sleep
            Sleeping for 5 sec
            [Pipeline] echo
            Sleep: 5
            
            Show
            h0riz0n Brent Goodacre added a comment - - edited I've noticed this problem as well, it really makes "Prepare for shutdown" quite useless for us because it's tedious to have to hard kill our currently running jobs. The whole idea of "Prepare for shutdown" is to allow us to finish the currently running jobs, not freeze them where they are. It's worth calling out that this has only happened in a recent upgrade of either Jenkins or one of the Pipeline plugins (Sorry it's vague but I'd estimate a change in the last 1-2 months) Here is a workflow script to repro: 1. Create and start a job that uses the following code: node( "testnode" ) { for ( int i = 0; i < 10; i++ ) { println "Sleep: ${i+1}" sleep(5) } } 2. Activate "Prepare for Shutdown" mode while the job is running. Expected Result: Job continues to run as it's already midway through execution. Actual Result: Job freezes Steven Walter this might not help you, but if you cancel "Prepare for Shutdown" and then on your job click "Pause/resume" to toggle the job into Paused, then Resumed states, the job will continue. Eg. Sleep: 3 [Pipeline] sleep Sleeping for 5 sec Pausing Resuming [Pipeline] echo Sleep: 4 [Pipeline] sleep Sleeping for 5 sec [Pipeline] echo Sleep: 5
            Hide
            bksaville Brian Saville added a comment - - edited

            To me, #3 is really the issue here. Due to the nature of pipeline jobs (serialized, stoppable at just about any point), it doesn't bother me that #2 happens. But cancelling the shutdown mode (without a restart) should cause the jobs to start back up again, and it currently doesn't.

            And it may go without saying, but we're seeing this issue too.

            Show
            bksaville Brian Saville added a comment - - edited To me, #3 is really the issue here. Due to the nature of pipeline jobs (serialized, stoppable at just about any point), it doesn't bother me that #2 happens. But cancelling the shutdown mode (without a restart) should cause the jobs to start back up again, and it currently doesn't. And it may go without saying, but we're seeing this issue too.
            Hide
            jglick Jesse Glick added a comment -

            An effect of the fix of JENKINS-32015.

            Show
            jglick Jesse Glick added a comment - An effect of the fix of JENKINS-32015 .
            Hide
            jglick Jesse Glick added a comment -

            Freezing the build—that is, the Groovy part—is deliberate, to prevent far worse problems, since we do not know when the actual JVM exit is going to come (TBD if we can safely block TermMilestone.COMPLETED). But the possibility that this mode would subsequently be cancelled was never considered. Pause + Resume is the appropriate workaround.

            The whole idea of "Prepare for shutdown" is to allow us to finish the currently running jobs

            It is to allow you to finish currently running freestyle (Maven, matrix, …) builds. So if you /safeRestart Jenkins will restart as soon as any of those are completed, and running Pipeline builds will be left alone.

            Show
            jglick Jesse Glick added a comment - Freezing the build—that is, the Groovy part—is deliberate, to prevent far worse problems, since we do not know when the actual JVM exit is going to come (TBD if we can safely block TermMilestone.COMPLETED ). But the possibility that this mode would subsequently be cancelled was never considered. Pause + Resume is the appropriate workaround. The whole idea of "Prepare for shutdown" is to allow us to finish the currently running jobs It is to allow you to finish currently running freestyle (Maven, matrix, …) builds. So if you /safeRestart Jenkins will restart as soon as any of those are completed, and running Pipeline builds will be left alone.
            Hide
            jyoukhana John Youkhana added a comment -

            Any update on this issue? I have been seeing the same issue where pipeline jobs don't complete when in shutdown mode and the only way to stop the job is to do a hard kill.

            I'm using Jenkins 2.66 and the pipelines plugin is version 2.5

            Show
            jyoukhana John Youkhana added a comment - Any update on this issue? I have been seeing the same issue where pipeline jobs don't complete when in shutdown mode and the only way to stop the job is to do a hard kill. I'm using Jenkins 2.66 and the pipelines plugin is version 2.5
            Hide
            jglick Jesse Glick added a comment -

            Manual Pause & Resume might work around it; have not tried.

            Show
            jglick Jesse Glick added a comment - Manual Pause & Resume might work around it; have not tried.
            Hide
            paulyb Paul Brimicombe added a comment -

            Jesse Glick I've tested pause & resume workaround and it seems to work.

            Doing the following leads to a hanging pipeline:

            • Trigger pipeline build.
            • Put Jenkins into Quiet Down mode.
            • Cancel Quiet Down mode.

            But doing the following makes the build resume and complete as expected:

            • Trigger pipeline build.
            • Pause pipeline build.
            • Put Jenkins into Quiet Down mode.
            • Cancel Quiet Down mode.
            • Resume pipeline build.

            I've had to write a (horrible) script that pauses all pipeline builds before triggering Quiet Down mode (we do this when we back up the master configuration files) and then resumes them all again afterwards.

            Is a fix for this on the cards soon? If you give me some pointers I might be able to help with the implementation.

            Show
            paulyb Paul Brimicombe added a comment - Jesse Glick I've tested pause & resume workaround and it seems to work. Doing the following leads to a hanging pipeline: Trigger pipeline build. Put Jenkins into Quiet Down mode. Cancel Quiet Down mode. But doing the following makes the build resume and complete as expected: Trigger pipeline build. Pause pipeline build. Put Jenkins into Quiet Down mode. Cancel Quiet Down mode. Resume pipeline build. I've had to write a (horrible) script that pauses all pipeline builds before triggering Quiet Down mode (we do this when we back up the master configuration files) and then resumes them all again afterwards. Is a fix for this on the cards soon? If you give me some pointers I might be able to help with the implementation.
            Hide
            jglick Jesse Glick added a comment -

            Not sure what the fix would look like. There is no event fired when quiet mode is cancelled AFAIK, so you would need to set up some kind of polling using Timer. At any rate, sounds like it should be easy enough to write a functional test for.

            Show
            jglick Jesse Glick added a comment - Not sure what the fix would look like. There is no event fired when quiet mode is cancelled AFAIK, so you would need to set up some kind of polling using Timer . At any rate, sounds like it should be easy enough to write a functional test for.
            Hide
            paulyb Paul Brimicombe added a comment -

            Given that pipelines are now a central feature of Jenkins would we be able to add an event for cancelling quiet mode (if one doesn't exist)? I feel like there might be other processes that could be interested in this and it would make for a much cleaner solution.

            Show
            paulyb Paul Brimicombe added a comment - Given that pipelines are now a central feature of Jenkins would we be able to add an event for cancelling quiet mode (if one doesn't exist)? I feel like there might be other processes that could be interested in this and it would make for a much cleaner solution.
            Hide
            mcrooney mcrooney added a comment -

            Seeing this on Jenkins 2.73.2 as well. Very odd that simple pipeline jobs totally hang just by entering quiet mode. It seems to cause them to hang in an unrecoverable way. We use "Quiet Down" to restart for upgrades to Jenkins and plugins in a graceful way, but very ironic and problematic that it is behaving very ungracefully by causing any in progress pipelines to get stuck.

            Show
            mcrooney mcrooney added a comment - Seeing this on Jenkins 2.73.2 as well. Very odd that simple pipeline jobs totally hang just by entering quiet mode. It seems to cause them to hang in an unrecoverable way. We use "Quiet Down" to restart for upgrades to Jenkins and plugins in a graceful way, but very ironic and problematic that it is behaving very ungracefully by causing any in progress pipelines to get stuck.
            Hide
            yasko Yasen Terziivanov added a comment -

            Also a problem in Jenkins: 2.89.4, Pipeline: 2.5 

            Show
            yasko Yasen Terziivanov added a comment - Also a problem in Jenkins: 2.89.4, Pipeline: 2.5 
            Hide
            reinholdfuereder Reinhold Füreder added a comment -

            ad #2 pipeline stops/freezes instead of expected and documented (e.g. also in UI "Manage Jenkins > Prepare for Shutdown") finishing of job:

            • this is surprising and unexpected considering also the documentation, but of course makes sense as described by Jesse.
            • however, would it make sense to change that to e.g. finish execution of currently active stage like commented in JENKINS-49365 – and somehow signal that in the UI similar to current "(part)" suffix in "build executor status" to allow for gentle shutdown at well-defined points? And therefore, as mentioned in JENKINS-49365, also when docker.image(...).inside(...) is used...

            ad #3 pipeline does not wake up after canceling the shutdown mode:

            • yes, I can confirm: "pause" (also) after cancelling followed by "resume" works and would be an IMHO acceptable workaround – although I really would not mind if that would be "just" fixed, naively assuming that it is not that difficult to do?

            While (resumable, i.e. non-docker.image(...).inside(...)-dependent) pipelines resume fine after a restart, there is a remainder in "build executor status" that (after the pipeline has finished) requires the workaround to abort it (the finished pipeline) by pressing the 'x' that in turn then asks "Are you sure you want to abort null?":

            Show
            reinholdfuereder Reinhold Füreder added a comment - ad #2 pipeline stops/freezes instead of expected and documented (e.g. also in UI "Manage Jenkins > Prepare for Shutdown") finishing of job: this is surprising and unexpected considering also the documentation, but of course makes sense as described by Jesse. however, would it make sense to change that to e.g. finish execution of currently active stage like commented in JENKINS-49365 – and somehow signal that in the UI similar to current "(part)" suffix in "build executor status" to allow for gentle shutdown at well-defined points? And therefore, as mentioned in JENKINS-49365 , also when docker.image(...).inside(...) is used... ad #3 pipeline does not wake up after canceling the shutdown mode: yes, I can confirm: "pause" (also) after cancelling followed by "resume" works and would be an IMHO acceptable workaround – although I really would not mind if that would be "just" fixed, naively assuming that it is not that difficult to do? While (resumable, i.e. non- docker.image(...).inside(...) -dependent) pipelines resume fine after a restart, there is a remainder in "build executor status" that (after the pipeline has finished) requires the workaround to abort it (the finished pipeline) by pressing the 'x' that in turn then asks "Are you sure you want to abort null?":
            Hide
            svanoort Sam Van Oort added a comment -

            Closing in preference to the pre-existing issue that has the same result (sigh, still not resolved yet).

            Show
            svanoort Sam Van Oort added a comment - Closing in preference to the pre-existing issue that has the same result (sigh, still not resolved yet).
            Hide
            taylor01 David Taylor added a comment -

            Any updates would be very appreciated. This bug is causing a lot of problems for our build system

            Show
            taylor01 David Taylor added a comment - Any updates would be very appreciated. This bug is causing a lot of problems for our build system
            Hide
            totoroliu Rick Liu added a comment -

            This still happened to Jenkins v*2.138.4* LTS

            Show
            totoroliu Rick Liu added a comment - This still happened to Jenkins v*2.138.4* LTS
            Hide
            jglick Jesse Glick added a comment -

            (incorrect resolution)

            Show
            jglick Jesse Glick added a comment - (incorrect resolution)

              People

              • Assignee:
                Unassigned
                Reporter:
                srwalter Steven Walter
              • Votes:
                20 Vote for this issue
                Watchers:
                26 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: