Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-34256

Preparing Jenkins For Shutdown Hangs Running Pipelines

    Details

    • Similar Issues:
    • Released As:
      workflow-cps 2.78

      Description

      Start a couple long-running pipelines with

      node

      { sleep 100 }

      Queue up a few more jobs. Go to "manage jenkins" and "prepare for shutdown."

      Now pipeline jobs that would finish and unenqueue never finish and have to manually be killed (which does work). Freestyle jobs complete normally. Queued jobs aren't run, so that part of prepare-for-shutdown works.

      Even stranger: upon killing and restarting with Ctrl+C, we get this lovely conundrum:

      Those pipeline builds won't show up in the build queue on the main screen.

      Checks to do:

      • Regression in core?
      • Regression in pipeline?
      • does /safeRestart or /restart trigger it?

        Attachments

          Issue Links

            Activity

            Hide
            medianick Nick Jones added a comment -

            Devin Nusbaum could you clarify whether this same logic/behavior also applies to the restart that happens after plugin installation (when checking the "Restart Jenkins when installation is complete and no jobs are running" checkbox) or when clicking the Restart Safely button under Manage Jenkins (i.e., the /safeRestart URL, as enabled by https://plugins.jenkins.io/saferestart)? Do running pipeline jobs get paused in those circumstances too and now (with Pipeline: Groovy 2.78) automatically resumed once Jenkins is back up?

            Show
            medianick Nick Jones added a comment - Devin Nusbaum could you clarify whether this same logic/behavior also applies to the restart that happens after plugin installation (when checking the "Restart Jenkins when installation is complete and no jobs are running" checkbox) or when clicking the Restart Safely button under Manage Jenkins (i.e., the /safeRestart URL, as enabled by https://plugins.jenkins.io/saferestart)? Do running pipeline jobs get paused in those circumstances too and now (with Pipeline: Groovy 2.78) automatically resumed once Jenkins is back up?
            Hide
            dnusbaum Devin Nusbaum added a comment -

            Could you clarify whether this same logic/behavior also applies to the restart that happens after plugin installation (when checking the "Restart Jenkins when installation is complete and no jobs are running" checkbox) or when clicking the Restart Safely button under Manage Jenkins (i.e., the /safeRestart URL

            Both of these situations use the /safeRestart URL behind the scenes, which puts Jenkins into the same state as "Prepare for shutdown", which prevents new builds from being started and causes Pipeline builds to pause. The difference between /safeRestart and "Prepare for shutdown" is that safeRestart will also automatically restart Jenkins once all non-Pipeline jobs have completed and all Pipeline jobs have been paused, whereas "Prepare for shutdown" does not actually restart Jenkins.

            Even before Pipeline: Groovy version 2.78, once Jenkins restarted due to /safeRestart, all Pipelines should have resumed automatically, and they should continue to have that behavior in Pipeline: Groovy 2.78. If your Pipelines are not resuming after the restart, please open a new ticket, including steps to reproduce the issue from scratch and any messages from your Jenkins logs or Pipeline build logs that seem relevant.

            Show
            dnusbaum Devin Nusbaum added a comment - Could you clarify whether this same logic/behavior also applies to the restart that happens after plugin installation (when checking the "Restart Jenkins when installation is complete and no jobs are running" checkbox) or when clicking the Restart Safely button under Manage Jenkins (i.e., the /safeRestart URL Both of these situations use the /safeRestart URL behind the scenes, which puts Jenkins into the same state as "Prepare for shutdown", which prevents new builds from being started and causes Pipeline builds to pause. The difference between /safeRestart and "Prepare for shutdown" is that safeRestart will also automatically restart Jenkins once all non-Pipeline jobs have completed and all Pipeline jobs have been paused, whereas "Prepare for shutdown" does not actually restart Jenkins. Even before Pipeline: Groovy version 2.78, once Jenkins restarted due to /safeRestart , all Pipelines should have resumed automatically, and they should continue to have that behavior in Pipeline: Groovy 2.78. If your Pipelines are not resuming after the restart, please open a new ticket, including steps to reproduce the issue from scratch and any messages from your Jenkins logs or Pipeline build logs that seem relevant.
            Hide
            medianick Nick Jones added a comment -

            Thanks Devin Nusbaum. So the logic change in 2.78 is only for the specific situation where Jenkins is "put to sleep" (Prepare for Shutdown) and then "woken up" (Cancel Shutdown) without actually being restarted? I.e., the intended/expected behavior even prior to 2.78 is that paused pipeline builds would resume automatically after an actual service restart? I've definitely seen them not resume after a restart, so I'll endeavor to reproduce the problem and then file a new bug with details.

            Show
            medianick Nick Jones added a comment - Thanks Devin Nusbaum . So the logic change in 2.78 is only for the specific situation where Jenkins is "put to sleep" (Prepare for Shutdown) and then "woken up" (Cancel Shutdown) without actually being restarted? I.e., the intended/expected behavior even prior to 2.78 is that paused pipeline builds would resume automatically after an actual service restart? I've definitely seen them not resume after a restart, so I'll endeavor to reproduce the problem and then file a new bug with details.
            Hide
            dnusbaum Devin Nusbaum added a comment -

            So the logic change in 2.78 is only for the specific situation where Jenkins is "put to sleep" (Prepare for Shutdown) and then "woken up" (Cancel Shutdown) without actually being restarted? I.e., the intended/expected behavior even prior to 2.78 is that paused pipeline builds would resume automatically after an actual service restart?

            Yes, although note that you can also cancel /safeRestart before the restart happens, and the logic change fixes that case too.

            I've definitely seen them not resume after a restart, so I'll endeavor to reproduce the problem and then file a new bug with details.

            Ok, great!

            Show
            dnusbaum Devin Nusbaum added a comment - So the logic change in 2.78 is only for the specific situation where Jenkins is "put to sleep" (Prepare for Shutdown) and then "woken up" (Cancel Shutdown) without actually being restarted? I.e., the intended/expected behavior even prior to 2.78 is that paused pipeline builds would resume automatically after an actual service restart? Yes, although note that you can also cancel /safeRestart before the restart happens, and the logic change fixes that case too. I've definitely seen them not resume after a restart, so I'll endeavor to reproduce the problem and then file a new bug with details. Ok, great!
            Hide
            reinholdfuereder Reinhold Füreder added a comment -

            Devin Nusbaum I can confirm that your fix works really fine!

            Because – now some coughing and red face – I accidentally restarted Jenkins master without waiting for pipelines to complete (of course looking forward to JENKINS-60434): and there were some non-minor real world pipelines running... Just one of them failed due to JENKINS-49365...

            Show
            reinholdfuereder Reinhold Füreder added a comment - Devin Nusbaum I can confirm that your fix works really fine! Because – now some coughing and red face – I accidentally restarted Jenkins master without waiting for pipelines to complete (of course looking forward to JENKINS-60434 ): and there were some non-minor real world pipelines running... Just one of them failed due to JENKINS-49365 ...

              People

              • Assignee:
                dnusbaum Devin Nusbaum
                Reporter:
                svanoort Sam Van Oort
              • Votes:
                44 Vote for this issue
                Watchers:
                70 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: