Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-6473

Deadlock when rendering test result graphs

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Won't Fix
    • Component/s: core
    • Labels:
      None
    • Environment:
      JDK 1.6.0.17
      Red Hat Enterprise Linux Server release 5.4 (Tikanga)
      2.6.18-164.9.1.el5 #1 SMP Wed Dec 9 03:29:54 EST 2009 i686 i686 i386 GNU/Linux
      Hudson 1.356
    • Similar Issues:

      Description

      The attached threaddump shows a deadlock which was initiated when attempting to view a job in my Hudson instance.

      The deadlock is at the following two threads (these are snippets from the attachment):

      "Handling GET /view/Quicksilver/job/users-service/test/trend : http-8080-9":
      at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1394)

      • waiting to lock <0xadf7e550> (a java.lang.String)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1361)
        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
      • locked <0x700966f0> (a org.apache.catalina.loader.WebappClassLoader)
        at hudson.tasks.test.AbstractTestResultAction.createChart(AbstractTestResultAction.java:260)
        at hudson.tasks.test.AbstractTestResultAction.doGraph(AbstractTestResultAction.java:208)

      "Handling GET /view/Quicksilver/job/users-service/tasks/trendGraph/png : http-8080-10":
      at java.lang.ClassLoader.checkCerts(ClassLoader.java:745)

      • waiting to lock <0x700966f0> (a org.apache.catalina.loader.WebappClassLoader)
        at java.lang.ClassLoader.preDefineClass(ClassLoader.java:484)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:610)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
        at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2527)
      • locked <0xadf7e550> (a java.lang.String)
        at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1010)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1483)
      • locked <0xadf7e550> (a java.lang.String)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:296)
      • locked <0x6ff84938> (a hudson.ClassicPluginStrategy$DependencyClassLoader)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:296)
      • locked <0x6ff83078> (a java.net.URLClassLoader)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
      • locked <0x6ff83078> (a java.net.URLClassLoader)
        at hudson.plugins.analysis.graph.CategoryBuildResultGraph.createAreaChart(CategoryBuildResultGraph.java:391)
        at hudson.plugins.analysis.graph.PriorityGraph.createChart(PriorityGraph.java:55)

      I first started noticing this behavior when updating the Static Analysis Collector Plug-in, Checkstyle Plug-in, PMD Plug-in, and FindBugs Plug-in to the latest versions.

      I tried to copy and paste the list of my installed plugins and version numbers.

        Attachments

          Issue Links

            Activity

            Hide
            mindless Alan Harder added a comment -

            how repeatable is this? if you restart hudson and view the page again, does this happen every time?

            Threads from hudson core and an analysis plugin are both calling jfreechart at the same time, which both go to load some classes, apparently via classloaders coming in a different order. Looks like this is running in Tomcat? (what version?) Would be nice if tomcat was resilient to locking issues from loading via different classloaders.. maybe a different tomcat version or no tomcat would work better. Not sure what the best workaround in hudson code would be..

            Show
            mindless Alan Harder added a comment - how repeatable is this? if you restart hudson and view the page again, does this happen every time? Threads from hudson core and an analysis plugin are both calling jfreechart at the same time, which both go to load some classes, apparently via classloaders coming in a different order. Looks like this is running in Tomcat? (what version?) Would be nice if tomcat was resilient to locking issues from loading via different classloaders.. maybe a different tomcat version or no tomcat would work better. Not sure what the best workaround in hudson code would be..
            Hide
            mindless Alan Harder added a comment -

            I see synchronized (name.intern()) in Tomcat 6.0.26 code.. I don't see this in 6.0.20 (it's there from 21 on) or 5.5.29 or 7.0.0-RC1, so maybe you'll have better luck with those?

            Show
            mindless Alan Harder added a comment - I see synchronized (name.intern()) in Tomcat 6.0.26 code.. I don't see this in 6.0.20 (it's there from 21 on) or 5.5.29 or 7.0.0-RC1, so maybe you'll have better luck with those?
            Hide
            msbcode msbcode added a comment -

            > how repeatable is this? if you restart hudson and view the page again, does this happen every time?

            I'm not certain about "every time" but it has definitely happened more than once.

            > Looks like this is running in Tomcat? (what version?)

            I believe 6.0.26 (don't have access to the machine at the current moment). Will try to downgrade on Monday and get back to you.

            Show
            msbcode msbcode added a comment - > how repeatable is this? if you restart hudson and view the page again, does this happen every time? I'm not certain about "every time" but it has definitely happened more than once. > Looks like this is running in Tomcat? (what version?) I believe 6.0.26 (don't have access to the machine at the current moment). Will try to downgrade on Monday and get back to you.
            Hide
            msbcode msbcode added a comment -

            I switched from using Tomcat to host Hudson to simply running it within Winstone - no further deadlock problems.

            Show
            msbcode msbcode added a comment - I switched from using Tomcat to host Hudson to simply running it within Winstone - no further deadlock problems.
            Hide
            mindless Alan Harder added a comment -

            See comments 12-17 in apache bug 44041 .. sounds like Tomcat 6.0.27 might work better.

            Show
            mindless Alan Harder added a comment - See comments 12-17 in apache bug 44041 .. sounds like Tomcat 6.0.27 might work better.
            Hide
            kohsuke Kohsuke Kawaguchi added a comment -

            WebAppClassLoader source code in question for a reference.

            The dead lock is the inconsistent lock order between the interned class name and the WebappClassLoader itself. The loadClass method itself isn't synchronized, which enables String->WebAppClassLoader order, but when called from methods like loadClassInternal, the order becomes the other way around.

            A possible work around is to externally synchronize WebappClassLoader before calling loadClass. This can be done from our classloader, and it might reduce the risk, if not eliminate the problem.

            Show
            kohsuke Kohsuke Kawaguchi added a comment - WebAppClassLoader source code in question for a reference. The dead lock is the inconsistent lock order between the interned class name and the WebappClassLoader itself. The loadClass method itself isn't synchronized, which enables String->WebAppClassLoader order, but when called from methods like loadClassInternal, the order becomes the other way around. A possible work around is to externally synchronize WebappClassLoader before calling loadClass. This can be done from our classloader, and it might reduce the risk, if not eliminate the problem.

              People

              • Assignee:
                Unassigned
                Reporter:
                msbcode msbcode
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: