Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-73051

thread-deadlock causing Jenkins hang

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • kubernetes-plugin
    • None
    • Jenkins version: 2.414.1
      Kuberntes-plugin version: 4029.v5712230ccb_f8
      K8S version: Azure K8S Service V1.28.3

      deadlock happens frequently and always happens when a build trigger concurrently many times,

      Jenkins cannot response to build trigger or build console output when error occurs

      only a restart to jenkins might help 

       

      Jenkins logs when error
      Some health checks are reporting as unhealthy: [thread-deadlock : [jenkins.util.Timer 3 locked on java.util.concurrent.locks.ReentrantLock$NonfairSync@4c9b7087 (owned by jenkins.util.Timer 8): at java.base@11.0.20/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.20/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.20/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885) at java.base@11.0.20/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:917) at java.base@11.0.20/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1240) at java.base@11.0.20/java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:267) at hudson.model.Queue.save(Queue.java:459) at hudson.model.Queue$Saver.run(Queue.java:3140) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67) at java.base@11.0.20/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base@11.0.20/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base@11.0.20/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base@11.0.20/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base@11.0.20/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11.0.20/java.lang.Thread.run(Thread.java:829) , Computer.threadPoolForRemoting 7 locked on java.util.concurrent.locks.ReentrantLock$NonfairSync@4c9b7087 (owned by jenkins.util.Timer 8): at java.base@11.0.20/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.20/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.20/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885) at java.base@11.0.20/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:917) at java.base@11.0.20/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1240) at java.base@11.0.20/java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:267) at hudson.model.Queue._withLock(Queue.java:1456) at hudson.model.Queue.withLock(Queue.java:1314) at jenkins.model.Nodes.updateNode(Nodes.java:201) at jenkins.model.Jenkins.updateNode(Jenkins.java:2252) at hudson.model.Node.save(Node.java:143) at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:247) at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:297) at hudson.slaves.SlaveComputer$$Lambda$481/0x0000000100c00040.call(Unknown Source) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80) at java.base@11.0.20/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base@11.0.20/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base@11.0.20/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11.0.20/java.lang.Thread.run(Thread.java:829) , jenkins.util.Timer 8 locked on org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher@4a90666a (owned by Computer.threadPoolForRemoting 7): at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.isLaunchSupported(KubernetesLauncher.java:91) at hudson.slaves.SlaveComputer.isLaunchSupported(SlaveComputer.java:247) at org.jenkinsci.plugins.durabletask.executors.OnceRetentionStrategy.check(OnceRetentionStrategy.java:81) at org.jenkinsci.plugins.durabletask.executors.OnceRetentionStrategy.check(OnceRetentionStrategy.java:46) at hudson.slaves.ComputerRetentionWork$1.run(ComputerRetentionWork.java:71) at hudson.model.Queue._withLock(Queue.java:1397) at hudson.model.Queue.withLock(Queue.java:1271) at hudson.slaves.ComputerRetentionWork.doRun(ComputerRetentionWork.java:62) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:92) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67) at java.base@11.0.20/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base@11.0.20/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base@11.0.20/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base@11.0.20/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base@11.0.20/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11.0.20/java.lang.Thread.run(Thread.java:829) ]]

        1. job.png
          job.png
          69 kB
        2. jstack.log
          144 kB

            henry_xiao henry
            henry_xiao henry
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: