Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-41127

Multiple pipeline instances running concurrently when concurrent execution disabled

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: core
    • Labels:
    • Environment:
      Jenkins 2.7.1
      Jenkins 2.49
      Pipeline plugin 2.4
    • Similar Issues:
    • Released As:
      Jenkins 2.136

      Description

      I have configured a Jenkins pipeline to disable concurrent builds:

      properties([
          disableConcurrentBuilds()
      ])
      

      However, I have noticed on some occasions the next 2 builds are pulled from the pipeline's queue and executed concurrently. Why this occurs is not obvious at all.

        Attachments

          Issue Links

            Activity

            Hide
            dnusbaum Devin Nusbaum added a comment - - edited

            Ok, I think the issue with lostPendings and flyweight tasks is that we loop through executors but not oneOffExecutors, which is where flyweight tasks are executed.

            I will test out looping through both tomorrow to see if that fixes it.

            Show
            dnusbaum Devin Nusbaum added a comment - - edited Ok, I think the issue with lostPendings and flyweight tasks is that we loop through executors but not oneOffExecutors, which is where flyweight tasks are executed . I will test out looping through both tomorrow to see if that fixes it.
            Hide
            svanoort Sam Van Oort added a comment -

            If that fixes it, it will probably be a very welcome change.

            Show
            svanoort Sam Van Oort added a comment - If that fixes it, it will probably be a very welcome change.
            Hide
            dnusbaum Devin Nusbaum added a comment -

            PR is up: https://github.com/jenkinsci/jenkins/pull/3562. Still looking into creating a regression test for it. I verified the change by running the same reproduction case as Alex/Andrew. Previously, a concurrent build would occur after ~250-750 builds, but after my fix I was able to run 3200 builds without any of them running concurrently.

            Show
            dnusbaum Devin Nusbaum added a comment - PR is up: https://github.com/jenkinsci/jenkins/pull/3562 . Still looking into creating a regression test for it. I verified the change by running the same reproduction case as Alex/Andrew. Previously, a concurrent build would occur after ~250-750 builds, but after my fix I was able to run 3200 builds without any of them running concurrently.
            Hide
            dnusbaum Devin Nusbaum added a comment -

            My best guess as to why this happens so infrequently is that normally after a call to Queue#maintain, the executor owning the flyweight task is the next thread that acquires the Queue's lock (in Executor#run), so Queue.pendings is cleared before the next call to Queue#maintain, but in the problematic case 2 calls to {Queue#maintain happen consecutively without Executor#run being executed yet, so the task is still in Queue.pendings in the second call to {Queue#maintain.

            I wonder if using a fair ordering policy for the Queue's lock would make this less likely, or if the Executor's run method isn't even waiting on the lock yet in the problematic case.

            Show
            dnusbaum Devin Nusbaum added a comment - My best guess as to why this happens so infrequently is that normally after a call to Queue#maintain , the executor owning the flyweight task is the next thread that acquires the Queue's lock (in Executor#run ), so Queue.pendings is cleared before the next call to Queue#maintain , but in the problematic case 2 calls to { Queue#maintain happen consecutively without Executor#run being executed yet, so the task is still in Queue.pendings in the second call to { Queue#maintain . I wonder if using a fair ordering policy for the Queue's lock would make this less likely, or if the Executor's run method isn't even waiting on the lock yet in the problematic case.
            Hide
            dnusbaum Devin Nusbaum added a comment -

            Fixed in Jenkins 2.136. I am marking this as an LTS candidate given the impact and simplicity of the fix, although we will have to give it some time to make sure there are no regressions.

            Show
            dnusbaum Devin Nusbaum added a comment - Fixed in Jenkins 2.136 . I am marking this as an LTS candidate given the impact and simplicity of the fix, although we will have to give it some time to make sure there are no regressions.

              People

              • Assignee:
                dnusbaum Devin Nusbaum
                Reporter:
                boon Joe Harte
              • Votes:
                3 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: