Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47860

Jobs waiting on input, grab an executor after restart

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      We have several jobs that use the "input" pipeline step. This step is not inside a node block so they usually do not grab an executor when waiting for user input, as expected.

      We often have many of these waiting over our 200+ jobs which is not normally an issue.

      However, upon restarting (e.g. after upgrading plugins), these jobs resume and take up an executor. They appear in the left hand side and have logs like as follows:

      Deploy to prod?
      Proceed or Abort
      Resuming build at Tue Nov 07 12:12:56 GMT 2017 after Jenkins restart
      Waiting to resume part of Folder Name » Job Name » master #47: ???
      Waiting to resume part of Folder Name » Job Name » master #47: Waiting for next available executor
      Waiting to resume part of Folder Name » Job Name » master #47: Waiting for next available executor
      Waiting to resume part of Folder Name » Job Name » master #47: Waiting for next available executor

      The current workaround is to manually abort each one but this time consuming and problematic if we do not want to abort them yet.

      Here is a possibly relevant portion of our shared library code, which gets called from most of our projects Jenkinsfiles:

      The relevant section of the code looks similar to this:

      for (def index = 0; index < allDeployEnvironments.size(); index++) {
        def nextEnvironment = ""
        if (index + 1 < allDeployEnvironments.size()) {
          nextEnvironment = allDeployEnvironments.get(index + 1)
        }
      
        //snip lines relating to deplpoyment.
      
        if (nextEnvironment) {   
          try {
            stage("${nextEnvironment} deployment prompt") {
              timeout(time: 30, unit: 'DAYS') {
              echo "Prompting for deploy to ${nextEnvironment}"
              input("Deploy to ${nextEnvironment}?")
            }
          }
        } catch (Exception ex) {
          echo "Input timed out"
          currentBuild.result = 'ABORTED'
          break
        }
      }

        Attachments

          Activity

          Hide
          abayer Andrew Bayer added a comment -

          Can you include your Jenkinsfile please?

          Show
          abayer Andrew Bayer added a comment - Can you include your Jenkinsfile please?
          Hide
          nicholasboll Nicholas Boll added a comment -

          Same happens to us. input step not in a node block and works fine normally but on Jenkins restart takes a node executor

          Show
          nicholasboll Nicholas Boll added a comment - Same happens to us. input step not in a node block and works fine normally but on Jenkins restart takes a node executor
          Hide
          rupert Rupert Madden-Abbott added a comment - - edited

          Unfortunately, its been a bit difficult for me to recreate this issue.

          It's not a simple matter of providing a Jenkinsfile because the Jenkinsfiles reference a shared library which is where the input step is called.

          I've added a snippet to the ticket which should add more context, but strips back some of the hopefully irrelevant complexity.

          Show
          rupert Rupert Madden-Abbott added a comment - - edited Unfortunately, its been a bit difficult for me to recreate this issue. It's not a simple matter of providing a Jenkinsfile because the Jenkinsfiles reference a shared library which is where the input step is called. I've added a snippet to the ticket which should add more context, but strips back some of the hopefully irrelevant complexity.
          Hide
          cecaron_mn Cédric Caron added a comment -

          We are facing the same issue.

          JENKINS-47860 and JENKINS-41791 currently make a CD pipeline based on Jenkins very brittle.

          Could those 2 bugs be moved to a higher level of criticity? 

          Show
          cecaron_mn Cédric Caron added a comment - We are facing the same issue. JENKINS-47860  and JENKINS-41791 currently make a CD pipeline based on Jenkins very brittle. Could those 2 bugs be moved to a higher level of criticity? 
          Hide
          rupert Rupert Madden-Abbott added a comment -

          Cédric Caron - are you able to provide a Jenkinsfile that replicates this issue? Part of the problem I've faced in trying to debug this issue is that I've not been able to replicate it reliably.

          Show
          rupert Rupert Madden-Abbott added a comment - Cédric Caron - are you able to provide a Jenkinsfile that replicates this issue? Part of the problem I've faced in trying to debug this issue is that I've not been able to replicate it reliably.
          Hide
          cecaron_mn Cédric Caron added a comment -

          Rupert Madden-Abbott It s not that straightforward as we are using a library that manage part of the pipeline but I will try to provide you something to reproduce.

          Show
          cecaron_mn Cédric Caron added a comment - Rupert Madden-Abbott  It s not that straightforward as we are using a library that manage part of the pipeline but I will try to provide you something to reproduce.
          Hide
          rupert Rupert Madden-Abbott added a comment -

          Cédric Caron - I completely understand - its exactly the same situation for us as well. It would be interesting to see what similarities there are with the snippet I posted in the original bug report above.

          Show
          rupert Rupert Madden-Abbott added a comment - Cédric Caron - I completely understand - its exactly the same situation for us as well. It would be interesting to see what similarities there are with the snippet I posted in the original bug report above.
          Hide
          arvid_ Arvid Arntzen added a comment -

          We experience the same issue with similar pipeline script containing input with timeout.

          It typically happens after reboot of master and the node that ran the last part of the pending builds waiting for input.

          All the builds waiting for input (that was not taking any executor before reboot) is now taking up all the executors on the node and are stuck there.

          Show
          arvid_ Arvid Arntzen added a comment - We experience the same issue with similar pipeline script containing input with timeout. It typically happens after reboot of master and the node that ran the last part of the pending builds waiting for input. All the builds waiting for input (that was not taking any executor before reboot) is now taking up all the executors on the node and are stuck there.

            People

            • Assignee:
              Unassigned
              Reporter:
              rupert Rupert Madden-Abbott
            • Votes:
              7 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated: