Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-22558

Deadlock condition when provisioning nodes and running the retention strategy to remove nodes

    Details

    • Similar Issues:

      Description

      There's a racing condition between node provisioning and the retention strategy to remove idle nodes.

      Using the threadDump, I retrieved the thread conditions when the deadlock happened:

      Here's the Retention Strategy Timer thread:

      jenkins.util.Timer [#6]
      
      "jenkins.util.Timer [#6]" Id=51 Group=main BLOCKED on hudson.model.Hudson@2fdebb41 owned by "Computer.threadPoolForRemoting [#1450]" Id=13155
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1706)
      	-  blocked on hudson.model.Hudson@2fdebb41
      	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
      	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
      	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@5e642b0a
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
      	at hudson.slaves.ComputerRetentionWork.doRun(ComputerRetentionWork.java:66)
      	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:744)
      
      	Number of locked synchronizers = 1
      

      Here's the Provivisioning thread:

      Computer.threadPoolForRemoting [#1450]
      
      "Computer.threadPoolForRemoting [#1450]" Id=13155 Group=main BLOCKED on hudson.plugins.ec2.EC2SpotRetentionStrategy@5e642b0a owned by "jenkins.util.Timer [#6]" Id=51
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:65)
      	-  blocked on hudson.plugins.ec2.EC2SpotRetentionStrategy@5e642b0a
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
      	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
      	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
      	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
      	-  locked java.lang.Object@7f564ca9
      	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
      	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
      	-  locked hudson.model.Hudson@2fdebb41
      	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
      	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
      	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@208a81be
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
      	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
      	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
      	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
      	-  locked java.lang.Object@7f564ca9
      	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
      	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
      	-  locked hudson.model.Hudson@2fdebb41
      	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
      	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
      	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@74c59c5f
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
      	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
      	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
      	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
      	-  locked java.lang.Object@7f564ca9
      	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
      	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
      	-  locked hudson.model.Hudson@2fdebb41
      	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
      	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
      	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@1e4b63aa
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
      	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
      	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
      	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
      	-  locked java.lang.Object@7f564ca9
      	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
      	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
      	-  locked hudson.model.Hudson@2fdebb41
      	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
      	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
      	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@edff3d5
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
      	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
      	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
      	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
      	-  locked java.lang.Object@7f564ca9
      	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
      	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
      	-  locked hudson.model.Hudson@2fdebb41
      	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
      	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
      	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@6f859bc2
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
      	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
      	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
      	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
      	-  locked java.lang.Object@7f564ca9
      	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
      	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
      	-  locked hudson.model.Hudson@2fdebb41
      	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
      	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
      	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@4b22eb23
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
      	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
      	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
      	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
      	-  locked java.lang.Object@7f564ca9
      	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
      	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
      	-  locked hudson.model.Hudson@2fdebb41
      	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
      	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
      	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@4f008c44
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
      	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
      	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
      	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
      	-  locked java.lang.Object@7f564ca9
      	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
      	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
      	-  locked hudson.model.Hudson@2fdebb41
      	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
      	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
      	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@252a5c30
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
      	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
      	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
      	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
      	-  locked java.lang.Object@7f564ca9
      	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
      	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
      	-  locked hudson.model.Hudson@2fdebb41
      	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
      	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
      	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@1c5a1b87
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
      	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
      	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
      	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
      	-  locked java.lang.Object@7f564ca9
      	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
      	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
      	-  locked hudson.model.Hudson@2fdebb41
      	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
      	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
      	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@360fa9ea
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
      	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
      	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
      	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
      	-  locked java.lang.Object@7f564ca9
      	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
      	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
      	-  locked hudson.model.Hudson@2fdebb41
      	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
      	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
      	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@241929b
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
      	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
      	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
      	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
      	-  locked java.lang.Object@7f564ca9
      	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
      	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
      	-  locked hudson.model.Hudson@2fdebb41
      	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
      	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
      	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@27f3ec80
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
      	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
      	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
      	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
      	-  locked java.lang.Object@7f564ca9
      	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
      	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
      	-  locked hudson.model.Hudson@2fdebb41
      	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
      	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
      	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@1d08a504
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
      	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
      	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
      	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
      	-  locked java.lang.Object@7f564ca9
      	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
      	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
      	-  locked hudson.model.Hudson@2fdebb41
      	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
      	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
      	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@c7c8313
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
      	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
      	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
      	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
      	-  locked java.lang.Object@7f564ca9
      	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
      	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
      	-  locked hudson.model.Hudson@2fdebb41
      	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
      	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
      	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@3e3587e8
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
      	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
      	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
      	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
      	-  locked java.lang.Object@7f564ca9
      	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
      	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
      	-  locked hudson.model.Hudson@2fdebb41
      	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
      	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
      	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@7a3ee5db
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
      	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
      	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
      	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
      	-  locked java.lang.Object@7f564ca9
      	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
      	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
      	-  locked hudson.model.Hudson@2fdebb41
      	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
      	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
      	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@202ba370
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
      	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
      	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
      	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
      	-  locked java.lang.Object@7f564ca9
      	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
      	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
      	-  locked hudson.model.Hudson@2fdebb41
      	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
      	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
      	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@2a3129be
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
      	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
      	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
      	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
      	-  locked java.lang.Object@7f564ca9
      	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
      	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
      	-  locked hudson.model.Hudson@2fdebb41
      	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
      	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
      	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@38738704
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
      	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
      	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
      	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
      	-  locked java.lang.Object@7f564ca9
      	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
      	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
      	-  locked hudson.model.Hudson@2fdebb41
      	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
      	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
      	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@c4748bb
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
      	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
      	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
      	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
      	-  locked java.lang.Object@7f564ca9
      	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
      	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
      	-  locked hudson.model.Hudson@2fdebb41
      	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
      	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
      	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@654666f2
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
      	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
      	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
      	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
      	-  locked java.lang.Object@7f564ca9
      	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
      	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
      	-  locked hudson.model.Hudson@2fdebb41
      	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
      	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
      	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@7d635e40
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
      	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
      	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
      	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
      	-  locked java.lang.Object@7f564ca9
      	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
      	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
      	-  locked hudson.model.Hudson@2fdebb41
      	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
      	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
      	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@35b25a1e
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
      	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
      	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
      	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
      	-  locked java.lang.Object@7f564ca9
      	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
      	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
      	-  locked hudson.model.Hudson@2fdebb41
      	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
      	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
      	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@e9d7bb2
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
      	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
      	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
      	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
      	-  locked java.lang.Object@7f564ca9
      	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
      	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
      	-  locked hudson.model.Hudson@2fdebb41
      	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
      	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
      	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@2712efe8
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
      	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
      	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
      	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
      	-  locked java.lang.Object@7f564ca9
      	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
      	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
      	-  locked hudson.model.Hudson@2fdebb41
      	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
      	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
      	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@712322b
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
      	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
      	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
      	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
      	-  locked java.lang.Object@7f564ca9
      	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
      	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
      	-  locked hudson.model.Hudson@2fdebb41
      	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
      	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
      	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@5fbd1834
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
      	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
      	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
      	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
      	-  locked java.lang.Object@7f564ca9
      	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
      	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
      	-  locked hudson.model.Hudson@2fdebb41
      	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
      	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
      	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@57068640
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
      	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
      	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
      	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
      	-  locked java.lang.Object@7f564ca9
      	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
      	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
      	-  locked hudson.model.Hudson@2fdebb41
      	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
      	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
      	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@bd12c5d
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
      	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
      	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
      	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
      	-  locked java.lang.Object@7f564ca9
      	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
      	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
      	-  locked hudson.model.Hudson@2fdebb41
      	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
      	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
      	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@34b01559
      	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
      	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
      	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
      	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
      	-  locked java.lang.Object@7f564ca9
      	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
      	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
      	at jenkins.model.Jenkins.addNode(Jenkins.java:1699)
      	-  locked hudson.model.Hudson@2fdebb41
      	at hudson.plugins.ec2.EC2Cloud$1.call(EC2Cloud.java:360)
      	at hudson.plugins.ec2.EC2Cloud$1.call(EC2Cloud.java:355)
      	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:744)
      
      	Number of locked synchronizers = 1
      	- java.util.concurrent.ThreadPoolExecutor$Worker@126cd40b
      

        Attachments

          Issue Links

            Activity

            Hide
            recampbell Ryan Campbell added a comment -

            I've submitted a pull request to fix this: https://github.com/jenkinsci/ec2-plugin/pull/89

            Show
            recampbell Ryan Campbell added a comment - I've submitted a pull request to fix this: https://github.com/jenkinsci/ec2-plugin/pull/89
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Ryan Campbell
            Path:
            src/main/java/hudson/plugins/ec2/EC2RetentionStrategy.java
            http://jenkins-ci.org/commit/ec2-plugin/315c65ec432aaa02d3f7de77d74f4ad36dc191fc
            Log:
            JENKINS-22558 Avoid deadlocks by using RentrantLock#tryLock() instead of synchronized.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Ryan Campbell Path: src/main/java/hudson/plugins/ec2/EC2RetentionStrategy.java http://jenkins-ci.org/commit/ec2-plugin/315c65ec432aaa02d3f7de77d74f4ad36dc191fc Log: JENKINS-22558 Avoid deadlocks by using RentrantLock#tryLock() instead of synchronized.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Francis Upton
            Path:
            src/main/java/hudson/plugins/ec2/EC2RetentionStrategy.java
            http://jenkins-ci.org/commit/ec2-plugin/fb4f3fa27f63734e007f434fae8f051eeee16f1d
            Log:
            Merge pull request #89 from recampbell/JENKINS-22558

            JENKINS-22558 Avoid deadlocks by using RentrantLock#tryLock() instead of synchronized.

            Compare: https://github.com/jenkinsci/ec2-plugin/compare/68631a44c1c5...fb4f3fa27f63

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Francis Upton Path: src/main/java/hudson/plugins/ec2/EC2RetentionStrategy.java http://jenkins-ci.org/commit/ec2-plugin/fb4f3fa27f63734e007f434fae8f051eeee16f1d Log: Merge pull request #89 from recampbell/ JENKINS-22558 JENKINS-22558 Avoid deadlocks by using RentrantLock#tryLock() instead of synchronized. Compare: https://github.com/jenkinsci/ec2-plugin/compare/68631a44c1c5...fb4f3fa27f63

              People

              • Assignee:
                francisu Francis Upton
                Reporter:
                taksan taksan
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: