Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46494

Certain conditions might lock more resources than required quantity

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Labels:
      None
    • Environment:
      CloudBees Jenkins Enterprise 2.32.2.1-rolling
      lockable resources plugin version 1.x & 2.x
    • Similar Issues:

      Description

      In strained situations, more resources than you asked for might get locked. This creates a bottleneck problem related to resources but it cannot be solved by adding more resources.

      In worst cases will always lock all resources with a certain label instead of the required amount.

      You can recreate the issue by configuring a couple (say 4) of lockable resources with a tag. Trigger a bunch of these jobs with NUM_RESOURCES between 1 - 4 to build up a queue of jobs waiting for their resources to become available. Eventually the error will start appearing.

      properties([
          parameters ([
              string(name: 'NUM_RESOURCES')
          ])
      ]);
      
      node {
          def askFor = params.NUM_RESOURCES as Integer
          lock(label: "FOO", quantity: askFor) {
              def clients = org.jenkins.plugins.lockableresources.LockableResourcesManager.class.get().getResourcesFromBuild(currentBuild.getRawBuild())
              echo("Asked for ${askFor} resources, got ${clients.size()}: ${clients.toString()}")
              sleep (60 * 10)
              if (askFor != clients.size()) {
                  error ("Wrong amount of resources acquired")
              }
          }
      }
      

        Attachments

          Issue Links

            Activity

            Hide
            beyerj Jens Beyer added a comment -

            Seeing this issue massively.

            We have 9 labelled resources.

            Build 1 reserves 4.

            Build 2 reserves 4.

            Build 3 wants to reserve 4, and waits.

            Build 1 finishes and unlocks.

            Now the bug: Build 3 locks 5, although in log it says it locks 4.

             

            I'll happily play beta tester for a .hpi, this is bugging us quite much (we also often see builds reserve 4 to 5 when they only request 1, and this is really blocking) 

             

            Show
            beyerj Jens Beyer added a comment - Seeing this issue massively. We have 9 labelled resources. Build 1 reserves 4. Build 2 reserves 4. Build 3 wants to reserve 4, and waits. Build 1 finishes and unlocks. Now the bug: Build 3 locks 5, although in log it says it locks 4.   I'll happily play beta tester for a .hpi, this is bugging us quite much (we also often see builds reserve 4 to 5 when they only request 1, and this is really blocking)   
            Hide
            reinholdfuereder Reinhold Füreder added a comment - - edited

            I think I just stumbled over this issue as well and the change in linked pull request https://github.com/jenkinsci/lockable-resources-plugin/pull/66 actually looks reasonable/applicable at a quick naive glance...

            So here is my situation:

            1. Started playing around with lock based on 'label' (2 lockable resources with same label), but forgot to specify 'quantity' => so it reserved all available 2 resources (with this label)
            2. Fixing the problem by specifying 'quantity: 1'
            3. Triggering a new pipeline build with the fix, WHILE an old pipeline build WITHOUT THE FIX was still running AND was STILL LOCKING ALL 2 RESOURCES:
              • The old build (#793, 20180417-133734-revUNKNOWN) WITHOUT the fix:
                13:38:43 Trying to acquire lock on [Label: PhpStorm]
                13:38:43 Found 0 available resource(s). Waiting for correct amount: 2.
                13:38:43 [Label: PhpStorm] is locked, waiting...
                13:41:36 Lock acquired on [Label: PhpStorm]
                ...
                13:46:29 Lock released on resource [Label: PhpStorm]
                
              • The new build (#794, 20180417-134149-revUNKNOWN) WITH the fix and right (lying logs) and wrong behaviour:
                13:45:36 Trying to acquire lock on [Label: PhpStorm, Quantity: 1]
                13:45:36 Found 0 available resource(s). Waiting for correct amount: 1.
                13:45:36 [Label: PhpStorm, Quantity: 1] is locked, waiting...
                13:46:29 Lock acquired on [Label: PhpStorm, Quantity: 1]
                ...
                
            4. So the new pipeline build had to correctly wait for the releasing of the lock, but it then incorrectly locked again for all 2 resources instead of just 1:
              • Lockable Resources view:
            5. Subsequent builds – after a coincidental pause/break that seemingly allowed to release all locks and give them completely back, instead of re-using them for queued/waiting builds – seemed to behave correctly then (and just lock 1 resource)
            Show
            reinholdfuereder Reinhold Füreder added a comment - - edited I think I just stumbled over this issue as well and the change in linked pull request https://github.com/jenkinsci/lockable-resources-plugin/pull/66 actually looks reasonable/applicable at a quick naive glance... So here is my situation: Started playing around with lock based on 'label' (2 lockable resources with same label), but forgot to specify 'quantity' => so it reserved all available 2 resources (with this label) Fixing the problem by specifying 'quantity: 1' Triggering a new pipeline build with the fix, WHILE an old pipeline build WITHOUT THE FIX was still running AND was STILL LOCKING ALL 2 RESOURCES: The old build (#793, 20180417-133734-revUNKNOWN) WITHOUT the fix: 13:38:43 Trying to acquire lock on [Label: PhpStorm] 13:38:43 Found 0 available resource(s). Waiting for correct amount: 2. 13:38:43 [Label: PhpStorm] is locked, waiting... 13:41:36 Lock acquired on [Label: PhpStorm] ... 13:46:29 Lock released on resource [Label: PhpStorm] The new build (#794, 20180417-134149-revUNKNOWN) WITH the fix and right (lying logs) and wrong behaviour: 13:45:36 Trying to acquire lock on [Label: PhpStorm, Quantity: 1] 13:45:36 Found 0 available resource(s). Waiting for correct amount: 1. 13:45:36 [Label: PhpStorm, Quantity: 1] is locked, waiting... 13:46:29 Lock acquired on [Label: PhpStorm, Quantity: 1] ... So the new pipeline build had to correctly wait for the releasing of the lock, but it then incorrectly locked again for all 2 resources instead of just 1: Lockable Resources view: Subsequent builds – after a coincidental pause/break that seemingly allowed to release all locks and give them completely back, instead of re-using them for queued/waiting builds – seemed to behave correctly then (and just lock 1 resource)
            Hide
            reinholdfuereder Reinhold Füreder added a comment -

            Antonio Muñiz Since the github activities of Marcus Antonsson seem to indicate that he is unfortunately no longer active, could you (as one of the plugin owners) maybe take over this bug – in the hope that it is a low hanging fruit and the linked pull request actually makes really sense...

            Show
            reinholdfuereder Reinhold Füreder added a comment - Antonio Muñiz Since the github activities of Marcus Antonsson seem to indicate that he is unfortunately no longer active, could you (as one of the plugin owners) maybe take over this bug – in the hope that it is a low hanging fruit and the linked pull request actually makes really sense...
            Hide
            mantonsson Marcus Antonsson added a comment -

            Reinhold Füreder It's not a very active account but I'm still active.

            I've unassigned myself and stopped the progress for now (unsure about the workflow here). Someone with the right privilege need to review the pull request and merge it for release.

            Show
            mantonsson Marcus Antonsson added a comment - Reinhold Füreder It's not a very active account but I'm still active. I've unassigned myself and stopped the progress for now (unsure about the workflow here). Someone with the right privilege need to review the pull request and merge it for release.
            Hide
            reinholdfuereder Reinhold Füreder added a comment -

            Marcus Antonsson Sorry for my harsh words and thanks for the clarification... and of course thanks for the pull request...

            Show
            reinholdfuereder Reinhold Füreder added a comment - Marcus Antonsson Sorry for my harsh words and thanks for the clarification... and of course thanks for the pull request...

              People

              • Assignee:
                Unassigned
                Reporter:
                mantonsson Marcus Antonsson
              • Votes:
                4 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated: