Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-34450

Deadlock between Executor and AsynchronousExecution

    Details

    • Similar Issues:

      Description

      A deadlock has been observed sporadically during Pipeline-related tests:

      "jenkins.util.Timer [#3]":
        waiting for ownable synchronizer 0x00000006c25235e0, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
        which is held by "Executing ...SomeTest"
      "Executing ...SomeTest":
        waiting to lock monitor 0x00007fd8b8004328 (object 0x00000006c254f860, a org.jenkinsci.plugins.workflow.job.WorkflowRun$2),
        which is held by "jenkins.util.Timer [#3]"
      "jenkins.util.Timer [#3]":
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00000006c25235e0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
      	at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
      	at hudson.model.Executor.getCurrentWorkUnit(Executor.java:533)
      	at hudson.model.queue.WorkUnitContext.synchronizeEnd(WorkUnitContext.java:141)
      	at hudson.model.Executor.finish1(Executor.java:460)
      	at hudson.model.Executor.completedAsynchronous(Executor.java:484)
      	at jenkins.model.queue.AsynchronousExecution.completed(AsynchronousExecution.java:118)
      	- locked <0x00000006c254f860> (a org.jenkinsci.plugins.workflow.job.WorkflowRun$2)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun$3.run(WorkflowRun.java:249)
      	- locked <0x00000006c253fc30> (a java.util.concurrent.atomic.AtomicBoolean)
      	at ...
      "Executing ...SomeTest":
      	at jenkins.model.queue.AsynchronousExecution.getExecutor(AsynchronousExecution.java:98)
      	- waiting to lock <0x00000006c254f860> (a org.jenkinsci.plugins.workflow.job.WorkflowRun$2)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun$2.interrupt(WorkflowRun.java:234)
      	at hudson.model.Executor.interrupt(Executor.java:227)
      	at hudson.model.Executor.interrupt(Executor.java:193)
      	at hudson.model.Executor.interrupt(Executor.java:187)
      	at hudson.model.Executor.interrupt(Executor.java:173)
      	at hudson.model.Executor.doStop(Executor.java:867)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.doStop(WorkflowRun.java:629)
      	- locked <0x00000006c24f8f10> (a org.jenkinsci.plugins.workflow.job.WorkflowRun)
      	at ...SomeTest
      

      I think the bug is in core, not WorkflowRun. The timer thread is holding an AsynchronousExecution monitor and waiting for the corresponding Executor read lock. This is entirely happening in core code. The test thread is holding the executor write lock (from doStop) and waiting for the AsynchronousExecution monitor. doStop winds up calling back into AsynchronousExecution.interrupt, which in this case is calling getExecutor, as indeed the Javadoc recommends.

      (The test thread is also holding a lock on the WorkflowRun, and the timer thread a lock on WorkflowRun.completed, both of which are potentially concerning, but these do not seem involved here.)

        Attachments

          Issue Links

            Activity

            jglick Jesse Glick created issue -
            jglick Jesse Glick made changes -
            Field Original Value New Value
            Link This issue is blocking JENKINS-25938 [ JENKINS-25938 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "workflow-job PR 2 (Web Link)" [ 14242 ]
            jglick Jesse Glick made changes -
            Link This issue is related to JENKINS-31614 [ JENKINS-31614 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 170567 ] JNJira + In-Review [ 183958 ]
            abayer Andrew Bayer made changes -
            Labels deadlock executor workflow deadlock executor pipeline workflow
            abayer Andrew Bayer made changes -
            Labels deadlock executor pipeline workflow deadlock executor pipeline

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                jglick Jesse Glick
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: