Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50260

lock step interacts badly with timeout step

    Details

    • Similar Issues:

      Description

      I have a job which takes a lock {} inside of a timeout {} block.

      If the timeout expires, then the lock {} is not freed, which results in future taking of the lock becoming a deadlock.

      It seems like timeout {} should properly free the lock when forcing an exit from the lock() step? Is this just a bad pipeline/workflow pattern?

        Attachments

          Activity

          jekeller Jacob Keller created issue -
          Hide
          jekeller Jacob Keller added a comment -

           

          This is with Lockable Resource Plugin 2.0

          I'm not 100% sure if the issue is a timeout() killing the lock() or not. I looked at the pipeline graph, and I see a timeout set to 15 minutes and the lock is blocked for 13 minutes, so it's possible this isn't the issue. I also had 2 other branches of parallel pipeline that ended up blocking for 4 days before we killed the job.

          Specifically, the lock is only used inside a single build to limit parallelism of a section that cannot be run concurrently during a parallel{} chunk. Since the lockable resource name includes the BUILD_TAG, it should be a resource unique to the individual build, and thus the only places it should become locked are in other parallel threads.

          Any other ideas what might cause the deadlock?

          Show
          jekeller Jacob Keller added a comment -   This is with Lockable Resource Plugin 2.0 I'm not 100% sure if the issue is a timeout() killing the lock() or not. I looked at the pipeline graph, and I see a timeout set to 15 minutes and the lock is blocked for 13 minutes, so it's possible this isn't the issue. I also had 2 other branches of parallel pipeline that ended up blocking for 4 days before we killed the job. Specifically, the lock is only used inside a single build to limit parallelism of a section that cannot be run concurrently during a parallel{} chunk. Since the lockable resource name includes the BUILD_TAG, it should be a resource unique to the individual build, and thus the only places it should become locked are in other parallel threads. Any other ideas what might cause the deadlock?
          abayer Andrew Bayer made changes -
          Field Original Value New Value
          Component/s workflow-basic-steps-plugin [ 21712 ]
          Hide
          getrostt Tobias Getrost added a comment -

          It looks as if the timeout is the reason why the lock is not released. I created a test to reproduce the behaviour: https://github.com/getrostt/lockable-resources-plugin/tree/jenkins-50260

          Show
          getrostt Tobias Getrost added a comment - It looks as if the timeout is the reason why the lock is not released. I created a test to reproduce the behaviour:  https://github.com/getrostt/lockable-resources-plugin/tree/jenkins-50260
          Hide
          tomasbjerre Tomas Bjerre added a comment -

          I tried using:

          Build Timeout Plugin 1.19

          Lockable Resource Plugin 2.8-SNAPSHOT

          And with pipeline:

           

          lock('myresource') {
           println "locked"
           timeout(time: 2, unit: 'SECONDS') {
            println "sleep..."
            sleep 10
            println "sleep done !!"
           }
           println "leaving timeout"
          }
          println "after lock"
          timeout(time: 2, unit: 'SECONDS') {
           println "locking..."
           lock('myresource') {
            println "locked, sleeping..."
            sleep 10
            println "sleep done !!"
           }
           println "left lock"
          }
          println "left timeout"
          

          The log shows:

           

           

          Started by user unknown or anonymous
          Running in Durability level: MAX_SURVIVABILITY
          [Pipeline] Start of Pipeline
          [Pipeline] lock
          Trying to acquire lock on [myresource]
          Resource [myresource] did not exist. Created.
          Lock acquired on [myresource]
          [Pipeline] {
          [Pipeline] echo
          locked
          [Pipeline] timeout
          Timeout set to expire in 2 sec
          [Pipeline] {
          [Pipeline] echo
          sleep...
          [Pipeline] sleep
          Sleeping for 10 sec
          Cancelling nested steps due to timeout
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          Lock released on resource [myresource]
          [Pipeline] // lock
          [Pipeline] End of Pipeline
          Timeout has been exceeded
          Finished: ABORTED
          

           

           

          Would help if anyone can reproduce it with a sample pipeline and plugin versions.

          Show
          tomasbjerre Tomas Bjerre added a comment - I tried using: Build Timeout Plugin 1.19 Lockable Resource Plugin 2.8-SNAPSHOT And with pipeline:   lock( 'myresource' ) { println "locked" timeout(time: 2, unit: 'SECONDS' ) { println "sleep..." sleep 10 println "sleep done !!" } println "leaving timeout" } println "after lock" timeout(time: 2, unit: 'SECONDS' ) { println "locking..." lock( 'myresource' ) { println "locked, sleeping..." sleep 10 println "sleep done !!" } println "left lock" } println "left timeout" The log shows:     Started by user unknown or anonymous Running in Durability level: MAX_SURVIVABILITY [Pipeline] Start of Pipeline [Pipeline] lock Trying to acquire lock on [myresource] Resource [myresource] did not exist. Created. Lock acquired on [myresource] [Pipeline] { [Pipeline] echo locked [Pipeline] timeout Timeout set to expire in 2 sec [Pipeline] { [Pipeline] echo sleep... [Pipeline] sleep Sleeping for 10 sec Cancelling nested steps due to timeout [Pipeline] } [Pipeline] // timeout [Pipeline] } Lock released on resource [myresource] [Pipeline] // lock [Pipeline] End of Pipeline Timeout has been exceeded Finished: ABORTED     Would help if anyone can reproduce it with a sample pipeline and plugin versions.

            People

            • Assignee:
              Unassigned
              Reporter:
              jekeller Jacob Keller
            • Votes:
              4 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated: