Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-40879

Locked areas are executed multiple times in parallel

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      My pipeline consists of several exclusive sections like this, which should never be executed in parallel:

      milestone()
      lock(resource: "unit-tests", inversePrecedence: true) {
          milestone()
          stage('Unit Tests') {
          }
      }
      
      milestone()
      lock(resource: "integration-tests", inversePrecedence: true) {
          milestone()
          stage('Integration Tests') {
          }
      }
      

      This worked perfectly fine with lockable-resources-plugin 1.10.
      However with version 1.11 the locking does not work properly anymore.

      It happens that two builds are running the integration tests in parallel,
      although this should not be possible given the above configuration.

      Whenever this happens I have a look at the http://<jenkins>/lockable-resources overview.
      It shows that the resource is locked by only one build, although two builds are just concurrently executing the integration tests.

      I am afraid it has to do with the recently implemented features:

      Since I consider this a bug and regression, I changed the priority to critical.

        Attachments

          Issue Links

            Activity

            Hide
            jjpickens Jason Pickens added a comment -

            I am also experiencing this bug. I am running 3 instances of the same pipeline job in parallel (build1, build2, build3), all in contention for Resource 'A'. build1 acquires 'A', the next 2 wait. When the first job releases Resource A (via a Java exception in the groovy script), both build2 and build3 log that they have acquired the resource and execute the critical section in parallel. Chaos ensues.

            Show
            jjpickens Jason Pickens added a comment - I am also experiencing this bug. I am running 3 instances of the same pipeline job in parallel (build1, build2, build3), all in contention for Resource 'A'. build1 acquires 'A', the next 2 wait. When the first job releases Resource A (via a Java exception in the groovy script), both build2 and build3 log that they have acquired the resource and execute the critical section in parallel. Chaos ensues.
            Hide
            philmcardlecg Phil McArdle added a comment -

            I believe I hit the same issue. I have multiple stages queued to run in parallel, with some of them waiting on the same locks.

            My experience was that when any lock was released, all the locks seemed to be.

            Lock acquired on resource 1 {
                Lock acquired on resource 2 {
                }
                Lock acquired on resource 1 {
                }
            }
            

            In my case, a second lock was being acquired on resource 1 long before the log or the pipeline steps view showed it being released.

            I've downgraded to 1.10 and it runs correctly. I'm not using inversePrecedence, quantity or label.

            Show
            philmcardlecg Phil McArdle added a comment - I believe I hit the same issue. I have multiple stages queued to run in parallel, with some of them waiting on the same locks. My experience was that when any lock was released, all the locks seemed to be. Lock acquired on resource 1 { Lock acquired on resource 2 { } Lock acquired on resource 1 { } } In my case, a second lock was being acquired on resource 1 long before the log or the pipeline steps view showed it being released. I've downgraded to 1.10 and it runs correctly. I'm not using inversePrecedence, quantity or label.
            Hide
            abayer Andrew Bayer added a comment -

            PR up at https://github.com/jenkinsci/lockable-resources-plugin/pull/45, including a first commit with a test that seems to consistently reproduces the issue.

            Show
            abayer Andrew Bayer added a comment - PR up at https://github.com/jenkinsci/lockable-resources-plugin/pull/45 , including a first commit with a test that seems to consistently reproduces the issue.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Andrew Bayer
            Path:
            src/test/java/org/jenkins/plugins/lockableresources/LockStepTest.java
            http://jenkins-ci.org/commit/lockable-resources-plugin/19119d6cbb0af2057550d6886d1239b34d54b169
            Log:
            JENKINS-40879 Test to verify bug.

            This test seems to fail pretty consistently - not 100% sure it'll keep
            failing every time, but I haven't seen it pass yet except with a fix.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: src/test/java/org/jenkins/plugins/lockableresources/LockStepTest.java http://jenkins-ci.org/commit/lockable-resources-plugin/19119d6cbb0af2057550d6886d1239b34d54b169 Log: JENKINS-40879 Test to verify bug. This test seems to fail pretty consistently - not 100% sure it'll keep failing every time, but I haven't seen it pass yet except with a fix.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Andrew Bayer
            Path:
            src/main/java/org/jenkins/plugins/lockableresources/LockableResourcesManager.java
            http://jenkins-ci.org/commit/lockable-resources-plugin/a7cc1aa694beed617aae302a7e04e770c81ec7aa
            Log:
            [FIXED JENKINS-40879] Only call proceed with resources to lock.

            Previously, we were calling LockStepExecution.proceed with the full
            list of resources - which was resulting in everything getting
            unlocked if there was a next context, i.e., if anything else was
            waiting on a resource to be unlocked. That was wrong, obviously.

            This changes to instead only call proceed with the list of resource
            names we actually need to lock for the next context, which fixes the
            test in the previous commit.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: src/main/java/org/jenkins/plugins/lockableresources/LockableResourcesManager.java http://jenkins-ci.org/commit/lockable-resources-plugin/a7cc1aa694beed617aae302a7e04e770c81ec7aa Log: [FIXED JENKINS-40879] Only call proceed with resources to lock. Previously, we were calling LockStepExecution.proceed with the full list of resources - which was resulting in everything getting unlocked if there was a next context, i.e., if anything else was waiting on a resource to be unlocked. That was wrong, obviously. This changes to instead only call proceed with the list of resource names we actually need to lock for the next context, which fixes the test in the previous commit.

              People

              • Assignee:
                abayer Andrew Bayer
                Reporter:
                dawi Daniel Wilmer
              • Votes:
                4 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: