Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-8929

Deadlock when upstream and downstream jobs are blocked on each other

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Component/s: core
    • Labels:
      None

      Description

      Job A (deploy to functional test) is upstream of job B (run functional tests).

      We need job A to block while its downstream job builds - otherwise a deployment while the functional tests are running will fail job B.

      We need job B to block while its upstream job builds - otherwise trying to run the tests while the deployment is running will fail.

      Unfortunately, if you configure A to block on downstream builds and B to block on upstream builds and A and B are scheduled at the same time they both block each other and are queued indefinitely, requiring manual intervention to clear the deadlock.

      In this situation job A should build and B should remain blocked until A completes.

        Activity

        Hide
        cbos Cees Bos added a comment -

        We upgrade to version 1.403, since we also faced this issue.
        But now the block does not work anymore in all situations.

        We see situations were this fails.
        If both upstream and downstream job are queue and the downstream project finishes, both project get started.

        Do I need to log a new issue for that?

        As soon as a upstream project is running the downstream gets waiting in the queue.

        Show
        cbos Cees Bos added a comment - We upgrade to version 1.403, since we also faced this issue. But now the block does not work anymore in all situations. We see situations were this fails. If both upstream and downstream job are queue and the downstream project finishes, both project get started. Do I need to log a new issue for that? As soon as a upstream project is running the downstream gets waiting in the queue.
        Hide
        mahoney Robert Elliot added a comment -

        My fault for not writing the test cases that would have shown Kohsuke why I did not make the change symmetrical. I wanted to but I haven't managed to get my head round the testing strategy for Jenkins yet.

        Show
        mahoney Robert Elliot added a comment - My fault for not writing the test cases that would have shown Kohsuke why I did not make the change symmetrical. I wanted to but I haven't managed to get my head round the testing strategy for Jenkins yet.
        Hide
        scm_issue_link SCM/JIRA link daemon added a comment -

        Code changed in jenkins
        User: Robert Elliot
        Path:
        core/src/main/java/hudson/model/AbstractProject.java
        core/src/main/java/hudson/model/Queue.java
        http://jenkins-ci.org/commit/jenkins/bddc4f2c1ae569dbf7af4bc73f8dcf667bf407bb
        Log:
        [FIXED JENKINS-8929] Changed the definition of a building downstream project to one that is Building, Waiting, Pending or Buildable
        but NOT blocked. This prevents the deadlock condition where upstream project A is blocked on downstream project B, which is
        blocked on upstream project A - now in this scenario B will remain blocked but A will build, eventually freeing up B.

        Show
        scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Robert Elliot Path: core/src/main/java/hudson/model/AbstractProject.java core/src/main/java/hudson/model/Queue.java http://jenkins-ci.org/commit/jenkins/bddc4f2c1ae569dbf7af4bc73f8dcf667bf407bb Log: [FIXED JENKINS-8929] Changed the definition of a building downstream project to one that is Building, Waiting, Pending or Buildable but NOT blocked. This prevents the deadlock condition where upstream project A is blocked on downstream project B, which is blocked on upstream project A - now in this scenario B will remain blocked but A will build, eventually freeing up B.
        Hide
        scm_issue_link SCM/JIRA link daemon added a comment -

        Code changed in jenkins
        User: Kohsuke Kawaguchi
        Path:
        core/src/main/java/hudson/model/AbstractProject.java
        core/src/main/java/hudson/model/Queue.java
        http://jenkins-ci.org/commit/jenkins/1eff6776e81756b5f15c0b599210e433d366f11d
        Log:
        JENKINS-8929 follow up changes

        • AbstractProject.getBuildingUpstream/getBuildingDownstream()
          should be symmetric.
        • It's unlikely for us to add another category of items,
          so reverted getUnblockedItems() implementation to the older one
          that was more efficient.
        • ItemList was meant to be internal, so keep it as such,
          at least for now.
        Show
        scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: core/src/main/java/hudson/model/AbstractProject.java core/src/main/java/hudson/model/Queue.java http://jenkins-ci.org/commit/jenkins/1eff6776e81756b5f15c0b599210e433d366f11d Log: JENKINS-8929 follow up changes AbstractProject.getBuildingUpstream/getBuildingDownstream() should be symmetric. It's unlikely for us to add another category of items, so reverted getUnblockedItems() implementation to the older one that was more efficient. ItemList was meant to be internal, so keep it as such, at least for now.
        Hide
        scm_issue_link SCM/JIRA link daemon added a comment -

        Code changed in jenkins
        User: Kohsuke Kawaguchi
        Path:
        changelog.html
        http://jenkins-ci.org/commit/jenkins/37f9f722fd15096066062931bfa264d8d8c31952
        Log:
        recording JENKINS-8929 fix

        Show
        scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html http://jenkins-ci.org/commit/jenkins/37f9f722fd15096066062931bfa264d8d8c31952 Log: recording JENKINS-8929 fix

          People

          • Assignee:
            mahoney Robert Elliot
            Reporter:
            mahoney Robert Elliot
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: