Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-15858

Jenkinks UI slow due to constant build record loading

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      I've noticed issues with jenkins being blocked for several minutes in recent versions (even before 1.490, it started probably like 10 versions in the past).

      Here's thread dump showing such deadlocks: https://gist.github.com/4109900

      Note that my home page is using the dashboard and several threads are blocked on:

      	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:638)
      	- waiting to lock <0x5e3afde0> (a hudson.model.RunMap)
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:621)
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.getById(AbstractLazyLoadRunMap.java:498)
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:472)
      	at hudson.model.AbstractProject.getNearestOldBuild(AbstractProject.java:1025)
      	at hudson.maven.MavenModuleSetBuild.getModuleLastBuilds(MavenModuleSetBuild.java:434)
      	at hudson.maven.MavenModuleSetBuild.getResult(MavenModuleSetBuild.java:189)
      	at hudson.model.Run.getIconColor(Run.java:640)
      	at hudson.plugins.view.dashboard.stats.StatBuilds.getBuildStat(StatBuilds.java:50)
      

        Attachments

          Issue Links

            Activity

            Hide
            kohsuke Kohsuke Kawaguchi added a comment -

            I looked at the additional thread dumps vmassol-20130625*.txt and I think Jesse's earlier analysis still applies — they are showing that while the UI thread is stuck, it's busy loading records from the disk.

            I see that three thread dumps involve two jobs "xwiki-platform Quality Checks" and "xwiki-platform". They are both Maven projects with lots of modules, so I can imagine that if the cache is cold, this can result in a considerable delay. When I access this instance, I experience that the first page load time of those two jobs are considerable, yet if I reload the page it renders quickly enough.

            Between these and the perm gen problem, I suspect that JVM in question simplify doesn't have enough heap size to keep the cache warm enough. Again, I'd love to see the heap dump that I requested above to see if there's something wasting the heap.

            Another thought that occurred to me is if it helps to provide an option to make the build records a strong reference, instead of the weak reference. It shifts the memory pressure from build records to other soft references, but for some users it might be an useful trade off.

            Show
            kohsuke Kohsuke Kawaguchi added a comment - I looked at the additional thread dumps vmassol-20130625*.txt and I think Jesse's earlier analysis still applies — they are showing that while the UI thread is stuck, it's busy loading records from the disk. I see that three thread dumps involve two jobs "xwiki-platform Quality Checks" and "xwiki-platform". They are both Maven projects with lots of modules, so I can imagine that if the cache is cold, this can result in a considerable delay. When I access this instance, I experience that the first page load time of those two jobs are considerable, yet if I reload the page it renders quickly enough. Between these and the perm gen problem, I suspect that JVM in question simplify doesn't have enough heap size to keep the cache warm enough. Again, I'd love to see the heap dump that I requested above to see if there's something wasting the heap. Another thought that occurred to me is if it helps to provide an option to make the build records a strong reference, instead of the weak reference. It shifts the memory pressure from build records to other soft references, but for some users it might be an useful trade off.
            Hide
            kohsuke Kohsuke Kawaguchi added a comment -

            Updated the title of the ticket to reflect the status.

            There's no dead lock involved.

            Show
            kohsuke Kohsuke Kawaguchi added a comment - Updated the title of the ticket to reflect the status. There's no dead lock involved.
            Show
            jglick Jesse Glick added a comment - @vmassol you can try https://buildhive.cloudbees.com/job/jenkinsci/job/dashboard-view-plugin/18/org.jenkins-ci.plugins$dashboard-view/artifact/org.jenkins-ci.plugins/dashboard-view/2.7-SNAPSHOT/dashboard-view-2.7-SNAPSHOT.hpi if you want to still use the Dashboard View plugin.
            Hide
            sogabe sogabe added a comment -

            fixed in 2.8

            Show
            sogabe sogabe added a comment - fixed in 2.8
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            test/src/main/java/org/jvnet/hudson/test/RunLoadCounter.java
            http://jenkins-ci.org/commit/jenkins-test-harness/9dc74d4132ad4e40874c638dc3f1773c770d87e0
            Log:
            Lazy-loading utility test class designed for fix of JENKINS-15858.
            Originally-Committed-As: 03ca63c62c1f3e20f405b655bbc610ed21931a86

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: test/src/main/java/org/jvnet/hudson/test/RunLoadCounter.java http://jenkins-ci.org/commit/jenkins-test-harness/9dc74d4132ad4e40874c638dc3f1773c770d87e0 Log: Lazy-loading utility test class designed for fix of JENKINS-15858 . Originally-Committed-As: 03ca63c62c1f3e20f405b655bbc610ed21931a86

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                vmassol Vincent Massol
              • Votes:
                12 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: