Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-8929

Deadlock when upstream and downstream jobs are blocked on each other

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: core
    • Labels:
      None
    • Similar Issues:

      Description

      Job A (deploy to functional test) is upstream of job B (run functional tests).

      We need job A to block while its downstream job builds - otherwise a deployment while the functional tests are running will fail job B.

      We need job B to block while its upstream job builds - otherwise trying to run the tests while the deployment is running will fail.

      Unfortunately, if you configure A to block on downstream builds and B to block on upstream builds and A and B are scheduled at the same time they both block each other and are queued indefinitely, requiring manual intervention to clear the deadlock.

      In this situation job A should build and B should remain blocked until A completes.

        Attachments

          Activity

          Hide
          cbos Cees Bos added a comment -

          We upgrade to version 1.403, since we also faced this issue.
          But now the block does not work anymore in all situations.

          We see situations were this fails.
          If both upstream and downstream job are queue and the downstream project finishes, both project get started.

          Do I need to log a new issue for that?

          As soon as a upstream project is running the downstream gets waiting in the queue.

          Show
          cbos Cees Bos added a comment - We upgrade to version 1.403, since we also faced this issue. But now the block does not work anymore in all situations. We see situations were this fails. If both upstream and downstream job are queue and the downstream project finishes, both project get started. Do I need to log a new issue for that? As soon as a upstream project is running the downstream gets waiting in the queue.
          Hide
          mahoney Robert Elliot added a comment -

          My fault for not writing the test cases that would have shown Kohsuke why I did not make the change symmetrical. I wanted to but I haven't managed to get my head round the testing strategy for Jenkins yet.

          Show
          mahoney Robert Elliot added a comment - My fault for not writing the test cases that would have shown Kohsuke why I did not make the change symmetrical. I wanted to but I haven't managed to get my head round the testing strategy for Jenkins yet.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Robert Elliot
          Path:
          core/src/main/java/hudson/model/AbstractProject.java
          core/src/main/java/hudson/model/Queue.java
          http://jenkins-ci.org/commit/jenkins/bddc4f2c1ae569dbf7af4bc73f8dcf667bf407bb
          Log:
          [FIXED JENKINS-8929] Changed the definition of a building downstream project to one that is Building, Waiting, Pending or Buildable
          but NOT blocked. This prevents the deadlock condition where upstream project A is blocked on downstream project B, which is
          blocked on upstream project A - now in this scenario B will remain blocked but A will build, eventually freeing up B.

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Robert Elliot Path: core/src/main/java/hudson/model/AbstractProject.java core/src/main/java/hudson/model/Queue.java http://jenkins-ci.org/commit/jenkins/bddc4f2c1ae569dbf7af4bc73f8dcf667bf407bb Log: [FIXED JENKINS-8929] Changed the definition of a building downstream project to one that is Building, Waiting, Pending or Buildable but NOT blocked. This prevents the deadlock condition where upstream project A is blocked on downstream project B, which is blocked on upstream project A - now in this scenario B will remain blocked but A will build, eventually freeing up B.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          core/src/main/java/hudson/model/AbstractProject.java
          core/src/main/java/hudson/model/Queue.java
          http://jenkins-ci.org/commit/jenkins/1eff6776e81756b5f15c0b599210e433d366f11d
          Log:
          JENKINS-8929 follow up changes

          • AbstractProject.getBuildingUpstream/getBuildingDownstream()
            should be symmetric.
          • It's unlikely for us to add another category of items,
            so reverted getUnblockedItems() implementation to the older one
            that was more efficient.
          • ItemList was meant to be internal, so keep it as such,
            at least for now.
          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: core/src/main/java/hudson/model/AbstractProject.java core/src/main/java/hudson/model/Queue.java http://jenkins-ci.org/commit/jenkins/1eff6776e81756b5f15c0b599210e433d366f11d Log: JENKINS-8929 follow up changes AbstractProject.getBuildingUpstream/getBuildingDownstream() should be symmetric. It's unlikely for us to add another category of items, so reverted getUnblockedItems() implementation to the older one that was more efficient. ItemList was meant to be internal, so keep it as such, at least for now.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          changelog.html
          http://jenkins-ci.org/commit/jenkins/37f9f722fd15096066062931bfa264d8d8c31952
          Log:
          recording JENKINS-8929 fix

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html http://jenkins-ci.org/commit/jenkins/37f9f722fd15096066062931bfa264d8d8c31952 Log: recording JENKINS-8929 fix

            People

            • Assignee:
              mahoney Robert Elliot
              Reporter:
              mahoney Robert Elliot
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: