Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-28136

Won't run either job when both queued in an upstream/downstream situation

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Critical
    • Resolution: Duplicate
    • Component/s: core
    • Labels:
      None
    • Environment:
      Centos 6.4, Jenkins 1.610 / 1.611
    • Similar Issues:

      Description

      Recently upgraded to Jenkins 1.610. Nothing that 2 jobs were running that not supposed to run at the same time as I have "Block build when upstream project is building" and "Block build when downstream project is building". When to first job in the stream and verified configs set. When I went to the first downstream job, and look in the "Advanced Project Options" section, there's no "Advanced" button to click to look at settings. What's weird is I know there are settings as I "Use custom workspace" and can see (when I try to run the job) that this setting is still in effect. Upgraded to 1.611 to see if fixed and same issues exists. I had also recently installed the "Build Blocker" plug-in and wondered if that had any interactions. But after uninstalling and restarting Jenkins, still have the same issues. As I can't have to coordinate some jobs, this is a blocker.

        Attachments

          Activity

          Hide
          dblondin Douglas Blondin added a comment -

          I found the Advanced button, it was hidden.

          But the issue still remains that Jenkins is not handling starting jobs correctly in an upstream/downstream situation.

          In abstract, here's my setup job A is dependent on section A (using SVN).
          If A runs, it will spawn job B. Job B can also run if section B (using SVN) changes. If job B runs, it spawns job C (which has no SVN dependencies).

          So here's the scenario changes comes into to SVN B, spawn job B, it completes and spawn job C. While job C is running, another changes comes into SVN B. That queues job B but won't run as job C is running. Still while C is running, a SVN changes comes into area A. That queues job A and doesn't run as job C is still running. This is all as expected.

          Now job C finishes. Jobs A and B are queued and neither one starts (and I caught this when both had been in queue for over 4 hours). If I manually try to start either, nothing happens and both remained queued.

          The only solution was to kill one job (in my case I killed B, since I knew after A ran it would run B anyways – and get B's SVN changes). Immediately upon killing B in the queued state, A started running. (and when it finished, B ran, finished, and ran C).

          So it looks to me like there's a bug in Jenkins 1.611 (and was in 1.610) – where if 2 jobs are queued that are in a upstream/downstream relationship and both are configured to not run if upstream/downstream is running, then it will start neither.

          FYI, I was on an old version of Jenkins (1.530) and this functionality was working just fine. I upgraded to 1.610 for other features and then later discovered this broke.

          Also, I have uninstalled Build Blocker plugin (and restarted Jenkins) and my above ABC scenario still exists. So I don't think Build Blocker plugin has anything to do with the issue, just a bug in Jenkins.

          Show
          dblondin Douglas Blondin added a comment - I found the Advanced button, it was hidden. But the issue still remains that Jenkins is not handling starting jobs correctly in an upstream/downstream situation. In abstract, here's my setup job A is dependent on section A (using SVN). If A runs, it will spawn job B. Job B can also run if section B (using SVN) changes. If job B runs, it spawns job C (which has no SVN dependencies). So here's the scenario changes comes into to SVN B, spawn job B, it completes and spawn job C. While job C is running, another changes comes into SVN B. That queues job B but won't run as job C is running. Still while C is running, a SVN changes comes into area A. That queues job A and doesn't run as job C is still running. This is all as expected. Now job C finishes. Jobs A and B are queued and neither one starts (and I caught this when both had been in queue for over 4 hours). If I manually try to start either, nothing happens and both remained queued. The only solution was to kill one job (in my case I killed B, since I knew after A ran it would run B anyways – and get B's SVN changes). Immediately upon killing B in the queued state, A started running. (and when it finished, B ran, finished, and ran C). So it looks to me like there's a bug in Jenkins 1.611 (and was in 1.610) – where if 2 jobs are queued that are in a upstream/downstream relationship and both are configured to not run if upstream/downstream is running, then it will start neither. FYI, I was on an old version of Jenkins (1.530) and this functionality was working just fine. I upgraded to 1.610 for other features and then later discovered this broke. Also, I have uninstalled Build Blocker plugin (and restarted Jenkins) and my above ABC scenario still exists. So I don't think Build Blocker plugin has anything to do with the issue, just a bug in Jenkins.
          Hide
          danielbeck Daniel Beck added a comment - - edited

          Note that the options are documented as (emphasis mine):

          When this option is checked, Jenkins will prevent the project from building when a dependency of this project is in the queue, or building.

          When this option is checked, Jenkins will prevent the project from building when a child of this project is in the queue, or building.

          So this appears to behave as designed (or at least as documented).

          Show
          danielbeck Daniel Beck added a comment - - edited Note that the options are documented as (emphasis mine): When this option is checked, Jenkins will prevent the project from building when a dependency of this project is in the queue, or building . When this option is checked, Jenkins will prevent the project from building when a child of this project is in the queue, or building . So this appears to behave as designed (or at least as documented).
          Hide
          dblondin Douglas Blondin added a comment -

          I'm not sure what documentation you're reading (appears to be perhaps the "?" pop-up) The checkbox under the "Advanced Projects Option" says "Block build when upstream project is building". I would suggest that if to block also if queued, then the regular text by the text box should say "Block build when upstream project is building or queued".

          Second, this is not the way that it worked in 1.530 as I have numerous time when multiple jobs in a stream are queued and one does start running. So when was the implementation changed to also block on queuing? why?

          Third, perhaps there are useful scenarios to block if simply queued (but none come to mind for my immediate usage).

          Fourth, OK, if this check box if not the way, how can I simply block on building and not worry about if multiple jobs are queued?

          My scenario is Job A builds and also produces stuff (e.g., headers) used by Job B. Job B builds. Job C deploys that artifacts. So I can't have any running at the same time (e.g., if B runs while A running, B may fail as A may not have yet produced needed items – or worse, might change needed items partway thru B's build). If the checkbox is not the way, what is the way?

          Show
          dblondin Douglas Blondin added a comment - I'm not sure what documentation you're reading (appears to be perhaps the "?" pop-up) The checkbox under the "Advanced Projects Option" says "Block build when upstream project is building". I would suggest that if to block also if queued, then the regular text by the text box should say "Block build when upstream project is building or queued". Second, this is not the way that it worked in 1.530 as I have numerous time when multiple jobs in a stream are queued and one does start running. So when was the implementation changed to also block on queuing? why? Third, perhaps there are useful scenarios to block if simply queued (but none come to mind for my immediate usage). Fourth, OK, if this check box if not the way, how can I simply block on building and not worry about if multiple jobs are queued? My scenario is Job A builds and also produces stuff (e.g., headers) used by Job B. Job B builds. Job C deploys that artifacts. So I can't have any running at the same time (e.g., if B runs while A running, B may fail as A may not have yet produced needed items – or worse, might change needed items partway thru B's build). If the checkbox is not the way, what is the way?
          Hide
          danielbeck Daniel Beck added a comment -

          I'm not sure what documentation you're reading (appears to be perhaps the "?" pop-up)

          Yep, inline help.

          The checkbox under the "Advanced Projects Option" says "Block build when upstream project is building". I would suggest that if to block also if queued, then the regular text by the text box should say "Block build when upstream project is building or queued".

          Needs a balance between short enough to be suitable as label, and being long enough to properly explain what it does. You're right that it's probably too short in this case.

          Second, this is not the way that it worked in 1.530 as I have numerous time when multiple jobs in a stream are queued and one does start running. So when was the implementation changed to also block on queuing? why?

          Checking the sources, it appears that it always should have behaved like this. Too many changes between the two releases to easily determine what was fixed and when, but it you could narrow it down I'll take another look. There's a chance it's related to a long standing queue concurrency bug that only recently was fixed.

          Third, perhaps there are useful scenarios to block if simply queued (but none come to mind for my immediate usage).

          Many features are best left to plugins to allow users to customize and configure Jenkins as much as they want while not overloading the default UI out of the box. Plugins could certainly achieve this with the existing extension points.

          Fourth, OK, if this check box if not the way, how can I simply block on building and not worry about if multiple jobs are queued?

          The jenkinsci-users mailing list can best provide answers to your Jenkins usage questions.

          Show
          danielbeck Daniel Beck added a comment - I'm not sure what documentation you're reading (appears to be perhaps the "?" pop-up) Yep, inline help. The checkbox under the "Advanced Projects Option" says "Block build when upstream project is building". I would suggest that if to block also if queued, then the regular text by the text box should say "Block build when upstream project is building or queued". Needs a balance between short enough to be suitable as label, and being long enough to properly explain what it does. You're right that it's probably too short in this case. Second, this is not the way that it worked in 1.530 as I have numerous time when multiple jobs in a stream are queued and one does start running. So when was the implementation changed to also block on queuing? why? Checking the sources, it appears that it always should have behaved like this. Too many changes between the two releases to easily determine what was fixed and when, but it you could narrow it down I'll take another look. There's a chance it's related to a long standing queue concurrency bug that only recently was fixed. Third, perhaps there are useful scenarios to block if simply queued (but none come to mind for my immediate usage). Many features are best left to plugins to allow users to customize and configure Jenkins as much as they want while not overloading the default UI out of the box. Plugins could certainly achieve this with the existing extension points. Fourth, OK, if this check box if not the way, how can I simply block on building and not worry about if multiple jobs are queued? The jenkinsci-users mailing list can best provide answers to your Jenkins usage questions.
          Hide
          danielbeck Daniel Beck added a comment -

          Assuming this duplicates JENKINS-28926, for which a fix will be in Jenkins 1.618 or so.

          Show
          danielbeck Daniel Beck added a comment - Assuming this duplicates JENKINS-28926 , for which a fix will be in Jenkins 1.618 or so.

            People

            • Assignee:
              Unassigned
              Reporter:
              dblondin Douglas Blondin
            • Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: