Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-33671

all triggered downstream jobs blocked until all upstream jobs leave queue

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None

      We have Jenkins 1.596.2 and 2.25 of the Parameterized Trigger plugin on a system with 1 master node and 6 slave nodes that use a single executor each. We have 14 services that are built as Maven jobs, and each has a post build action that uses the parameterized trigger option to call another job that performs calls which update an external system used in our deployment and test framework (see trigger-screenshot). I want a post build action, and not a post-build step so that the build will not fail due to trouble with the triggered job. The downstream job is configured to allow concurrent builds, which it does when it is not blocked. Also it is not configured to block on the upstream job (see config-screenshot).

      This works great when only one of these services is getting built. However when we are at peak activity, there can be check-ins coming in so frequently that for an extended period one of those services will always be queued up to build. Then we observe the issue shown in the queue-screenshot where all of the triggered build jobs are blocked until every instance of the upstream jobs have left the queue, only then the triggered jobs will then unblock. Note the upstream jobs can still be building, just not in queue. Note also that frequently all of the blocked jobs will report they are blocked by the same upstream job, however when the build they all use the correct parameters from the actual jobs that triggered each of them. Other times the hover tip info for the jobs does show the unique parameters for each triggered job while they are queued up.

      This has the unfortunate side effect of introducing a substantial delay between the completion of a new service build and updating the external system with metadata about the new build. A small delay there is not a problem, but it can be as large as an hour and that is a problem. We don't really want to have to create a stand-alone service to listen for these calls, but if Jenkins introduces too much of a delay we may have. I have tried using a post-build step instead of a post-build-action for the triggering of the downstream job and the same behavior was seen.

        1. config-screenshot.png
          27 kB
          John McCullough
        2. queue-screenshot.png
          60 kB
          John McCullough
        3. trigger-screenshot.png
          29 kB
          John McCullough

            huybrechts huybrechts
            jmmccullough John McCullough
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: