Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-32622

deadlock due to support-core use of classloading during logging

    Details

    • Similar Issues:

      Description

      the support-core plugin causes classes to be loaded via remoting during logging.
      This can lead to an intresting deadlock as classload itself causes logging and you can get into a situation where the logging causes classloading - but the class loader needs to log but can't as it is blocked trying to load the class.

      support-core should not rely on libraries external to the plugin or the JDK.

      "pool-1-thread-1 for CLI connection to https://cloudbees.ci.cloudbees.com" prio=10 tid=0x00007f3f2c04d800 nid=0x6d6f waiting for monitor entry [0x00007f3f2b9b8000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at org.apache.commons.lang.time.FastDateFormat$PaddedNumberField.appendTo(FastDateFormat.java:1306)
      	at org.apache.commons.lang.time.FastDateFormat$PaddedNumberField.appendTo(FastDateFormat.java:1288)
      	at org.apache.commons.lang.time.FastDateFormat.applyRules(FastDateFormat.java:890)
      	at org.apache.commons.lang.time.FastDateFormat.format(FastDateFormat.java:821)
      	at com.cloudbees.jenkins.support.SupportLogFormatter.format(SupportLogFormatter.java:52)
      	at com.cloudbees.jenkins.support.SupportLogHandler.publish(SupportLogHandler.java:99)
      	at java.util.logging.Logger.log(Logger.java:616)
      	at java.util.logging.Logger.doLog(Logger.java:641)
      	at java.util.logging.Logger.log(Logger.java:730)
      	at hudson.remoting.Request$2.run(Request.java:344)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      
      "Channel reader thread: CLI connection to https://cloudbees.ci.cloudbees.com / waiting for hudson.remoting.Channel@64c53235:CLI connection to https://cloudbees.ci.cloudbees.com" prio=10 tid=0x00007f3f4c332000 nid=0x6d6e in Object.wait() [0x00007f3f2bab8000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000bb751e28> (a hudson.remoting.RemoteInvocationHandler$RPCRequest)
      	at hudson.remoting.Request.call(Request.java:146)
      	- locked <0x00000000bb751e28> (a hudson.remoting.RemoteInvocationHandler$RPCRequest)
      	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:182)
      	at com.sun.proxy.$Proxy2.fetch2(Unknown Source)
      	at hudson.remoting.DumbClassLoaderBridge.fetch2(DumbClassLoaderBridge.java:40)
      	at hudson.remoting.DumbClassLoaderBridge.fetch3(DumbClassLoaderBridge.java:52)
      	at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:157)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
      	- locked <0x00000000ce9a0078> (a hudson.remoting.RemoteClassLoader)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
      	at org.apache.commons.lang.time.FastDateFormat$PaddedNumberField.appendTo(FastDateFormat.java:1306)
      	at org.apache.commons.lang.time.FastDateFormat$PaddedNumberField.appendTo(FastDateFormat.java:1288)
      	at org.apache.commons.lang.time.FastDateFormat.applyRules(FastDateFormat.java:890)
      	at org.apache.commons.lang.time.FastDateFormat.format(FastDateFormat.java:821)
      	at com.cloudbees.jenkins.support.SupportLogFormatter.format(SupportLogFormatter.java:52)
      	at com.cloudbees.jenkins.support.SupportLogHandler.publish(SupportLogHandler.java:99)
      	at java.util.logging.Logger.log(Logger.java:616)
      	at java.util.logging.Logger.doLog(Logger.java:641)
      	at java.util.logging.Logger.log(Logger.java:730)
      	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:68)
      
      "main / waiting for hudson.remoting.Channel@64c53235:CLI connection to https://cloudbees.ci.cloudbees.com" prio=10 tid=0x00007f3f4c00a000 nid=0x6d63 in Object.wait() [0x00007f3f55cfc000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000ce9a0358> (a hudson.remoting.UserRequest)
      	at hudson.remoting.Request.call(Request.java:146)
      	- locked <0x00000000ce9a0358> (a hudson.remoting.UserRequest)
      	at hudson.remoting.Channel.call(Channel.java:752)
      	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:179)
      	at com.sun.proxy.$Proxy3.main(Unknown Source)
      	at hudson.cli.CLI.execute(CLI.java:336)
      	at hudson.cli.CLI._main(CLI.java:501)
      	at hudson.cli.CLI.main(CLI.java:387)
      

        Attachments

          Issue Links

            Activity

            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: James Nord
            Path:
            src/main/java/com/cloudbees/jenkins/support/SupportLogFormatter.java
            http://jenkins-ci.org/commit/support-core-plugin/bf508eb6535d3d907839f924062772fd9aa6cba7
            Log:
            JENKINS-32622 Switch to a ThreadLocal DateFormatter

            Attempt to avoid a deadlock observed in remoting by not causing any remote
            classes to be loaded when we are logging.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: James Nord Path: src/main/java/com/cloudbees/jenkins/support/SupportLogFormatter.java http://jenkins-ci.org/commit/support-core-plugin/bf508eb6535d3d907839f924062772fd9aa6cba7 Log: JENKINS-32622 Switch to a ThreadLocal DateFormatter Attempt to avoid a deadlock observed in remoting by not causing any remote classes to be loaded when we are logging.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Steven Christou
            Path:
            src/main/java/com/cloudbees/jenkins/support/SupportLogFormatter.java
            http://jenkins-ci.org/commit/support-core-plugin/d0c2ec7b8acd28f088483e3d6ffec5fb3a157db7
            Log:
            Merge pull request #55 from jtnord/JENKINS-32622

            JENKINS-32622 Switch to a ThreadLocal DateFormatter

            Compare: https://github.com/jenkinsci/support-core-plugin/compare/751b73fdff61...d0c2ec7b8acd

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Steven Christou Path: src/main/java/com/cloudbees/jenkins/support/SupportLogFormatter.java http://jenkins-ci.org/commit/support-core-plugin/d0c2ec7b8acd28f088483e3d6ffec5fb3a157db7 Log: Merge pull request #55 from jtnord/ JENKINS-32622 JENKINS-32622 Switch to a ThreadLocal DateFormatter Compare: https://github.com/jenkinsci/support-core-plugin/compare/751b73fdff61...d0c2ec7b8acd

              People

              • Assignee:
                teilo James Nord
                Reporter:
                teilo James Nord
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: