Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46738

Self-deadlock in ExecutorStepExecution$PlaceholderTask$PlaceholderExecutable$1.interrupt

    Details

    • Similar Issues:

      Description

      What is the business impact?

      Jenkins stop working due the deadlock between support-core plugin and core/org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask$PlaceholderExecutable$1.interrupt(ExecutorStepExecution.java:570)

      What is the incorrect behaviour?

      The deadlock should not happen

      What is the expected behaviour?

      You should be able to automatically create a bundle and stop a build execution

      Step by step to reproduce

      N/A

      Workaround

      Restart the instance, disable support-core plugin or avoid to stop builds. Issue does not seems to be very frequent.

      Regression

      I don't think this is a regression.

      Environment

      support-core:2.32:not-pinned
      Version: `2.7.21.1 (CloudBees Jenkins Enterprise 2.7.21.1-rolling)`

      Deadlock stacktrace

      Found one Java-level deadlock:
      =============================
      "SupportPlugin periodic bundle generator: writing support_2017-09-07_20.55.37.zip since Thu Sep 07 16:55:37 EDT 2017":
        waiting for ownable synchronizer 0x00000006f2ccb9e0, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
        which is held by "Handling POST /computer/(master)/executors/0/stop from XXXXXX : RequestHandlerThread[#155]"
      "Handling POST /computer/(master)/executors/0/stop from XXXXXX : RequestHandlerThread[#155]":
        waiting for ownable synchronizer 0x00000006f3362550, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
        which is held by "Handling POST /computer/(master)/executors/4/stop from XXXXX : RequestHandlerThread[#161]"
      "Handling POST /computer/(master)/executors/4/stop from XXXXXX : RequestHandlerThread[#161]":
        waiting for ownable synchronizer 0x00000006f41de5e0, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
        which is held by "Handling POST /computer/(master)/executors/6/stop from XXXXXXX : RequestHandlerThread[#162]"
      "Handling POST /computer/(master)/executors/6/stop from XXXXXXXX : RequestHandlerThread[#162]":
        waiting for ownable synchronizer 0x00000006f2ccb9e0, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
        which is held by "Handling POST /computer/(master)/executors/0/stop from XXXXXXX : RequestHandlerThread[#155]"
      
      Found 1 deadlock.
      
      
      

        Attachments

          Activity

          Hide
          jglick Jesse Glick added a comment -

          The support-core thread is innocent. We have three threads in a cyclic deadlock:

          "Handling POST /computer/(master)/executors/…/stop …":
          	at sun.misc.Unsafe.park(Native Method)
          	- parking to wait for  <…> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
          	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
          	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
          	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
          	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
          	at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
          	at hudson.model.Executor.getCurrentExecutable(Executor.java:503)
          	at hudson.model.Executor.of(Executor.java:966)
          	at hudson.model.Run.getExecutor(Run.java:517)
          	at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask$PlaceholderExecutable$1.interrupt(ExecutorStepExecution.java:570)
          	at hudson.model.Executor.interrupt(Executor.java:227)
          	at hudson.model.Executor.interrupt(Executor.java:197)
          	at hudson.model.Executor.interrupt(Executor.java:187)
          	at hudson.model.Executor.interrupt(Executor.java:173)
          	at hudson.model.Executor.doStop(Executor.java:864)
          

          Executor.interrupt acquires a per-Executor lock, then calls into foreign code (AsynchronousExecution.interrupt), which in this case is calling Executor.of, attempting to acquire the locks of other executors.

          Show
          jglick Jesse Glick added a comment - The support-core thread is innocent. We have three threads in a cyclic deadlock: "Handling POST /computer/(master)/executors/…/stop …": at sun.misc.Unsafe.park(Native Method) - parking to wait for <…> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283) at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727) at hudson.model.Executor.getCurrentExecutable(Executor.java:503) at hudson.model.Executor.of(Executor.java:966) at hudson.model.Run.getExecutor(Run.java:517) at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask$PlaceholderExecutable$1.interrupt(ExecutorStepExecution.java:570) at hudson.model.Executor.interrupt(Executor.java:227) at hudson.model.Executor.interrupt(Executor.java:197) at hudson.model.Executor.interrupt(Executor.java:187) at hudson.model.Executor.interrupt(Executor.java:173) at hudson.model.Executor.doStop(Executor.java:864) Executor.interrupt acquires a per- Executor lock, then calls into foreign code ( AsynchronousExecution.interrupt ), which in this case is calling Executor.of , attempting to acquire the locks of other executors.
          Hide
          svanoort Sam Van Oort added a comment -

          Released with v2.15

          Show
          svanoort Sam Van Oort added a comment - Released with v2.15

            People

            • Assignee:
              jglick Jesse Glick
              Reporter:
              fbelzunc Félix Belzunce Arcos
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: