Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-16023

Lazy loading causes massive delays after a period of inactivity when loading dashboard

    Details

    • Similar Issues:

      Description

      After a period of inactivity or when the JVM decides to run GC loading the dashboard can easily take longer than 1-2 minutes. Please see attached stacktrace.

        Attachments

          Issue Links

            Activity

            Hide
            johno Johno Crawford added a comment -

            I suspect this is actually caused by the massive amount of build history we had for some jobs. After enabling fine logging on hudson.model.RunMap it was obvious which jobs were causing Jenkins to hang. I have now configured discard old builds for the jobs, will report back if it helped or not.

            Show
            johno Johno Crawford added a comment - I suspect this is actually caused by the massive amount of build history we had for some jobs. After enabling fine logging on hudson.model.RunMap it was obvious which jobs were causing Jenkins to hang. I have now configured discard old builds for the jobs, will report back if it helped or not.
            Hide
            johno Johno Crawford added a comment - - edited

            Purging the build history has helped a little, however when the JVM runs GC Jenkins is forced to reload all the build history again.

            Show
            johno Johno Crawford added a comment - - edited Purging the build history has helped a little, however when the JVM runs GC Jenkins is forced to reload all the build history again.
            Hide
            johno Johno Crawford added a comment - - edited

            Looks like hudson.model.Job.getLastFailedBuild(Job.java:780) is the expensive operation here if the job in question has a long build history with no failures as it will load all history or until it finds a failed job, unstable job etc..

            Show
            johno Johno Crawford added a comment - - edited Looks like hudson.model.Job.getLastFailedBuild(Job.java:780) is the expensive operation here if the job in question has a long build history with no failures as it will load all history or until it finds a failed job, unstable job etc..
            Hide
            vlatombe Vincent Latombe added a comment -

            <vlatombe> probably now that builds are fetched lazily, it may be worth to store some information at job level like lastSuccessFul/lastUnstableBuild/lastFailedBuild build numbers or timestamps
            [14:26] <Johnou> yes I agree, doesn't make sense to load all jobs until it finds that information
            [14:26] <Johnou> defeats the point of lazy init

            Show
            vlatombe Vincent Latombe added a comment - <vlatombe> probably now that builds are fetched lazily, it may be worth to store some information at job level like lastSuccessFul/lastUnstableBuild/lastFailedBuild build numbers or timestamps [14:26] <Johnou> yes I agree, doesn't make sense to load all jobs until it finds that information [14:26] <Johnou> defeats the point of lazy init
            Hide
            hanabishi Marcus Jacobsson added a comment -

            We are also having some problem with long load times after upgrading to a version with lazy loading. We are using getPreviousSuccessfulBuild quite frequently and we will disable the code that uses this operation and see if it helps.

            We have are running a Jenkins server with close to 1000 different jobs and noticed that we have a lot of hits on the root, that I think will list all builds. After looking at some statistics from JavaMelody we see that we have quite a lot of hits to the root and to cc.xml in the root. Could this force Jenkins to load unnecessary build data to the memory? And if so, is there a way to prevent this?
            Request, % of cumulative time, Hits, Mean time (ms)
            /cc.xml GET 11 79,903 21
            / GET 6 127 7,746 101,459

            Show
            hanabishi Marcus Jacobsson added a comment - We are also having some problem with long load times after upgrading to a version with lazy loading. We are using getPreviousSuccessfulBuild quite frequently and we will disable the code that uses this operation and see if it helps. We have are running a Jenkins server with close to 1000 different jobs and noticed that we have a lot of hits on the root, that I think will list all builds. After looking at some statistics from JavaMelody we see that we have quite a lot of hits to the root and to cc.xml in the root. Could this force Jenkins to load unnecessary build data to the memory? And if so, is there a way to prevent this? Request, % of cumulative time, Hits, Mean time (ms) /cc.xml GET 11 79,903 21 / GET 6 127 7,746 101,459
            Hide
            jglick Jesse Glick added a comment -

            The implementation of JENKINS-16089 should fix this for the lastSuccessfulBuild permalink (and corresponding view column). I am perplexed by why other Job.getLast*Build methods were not also converted to use Permalink.

            Show
            jglick Jesse Glick added a comment - The implementation of JENKINS-16089 should fix this for the lastSuccessfulBuild permalink (and corresponding view column). I am perplexed by why other Job.getLast*Build methods were not also converted to use Permalink .
            Hide
            jglick Jesse Glick added a comment -

            If my hypothesis is correct, the fix should be easy, but the tricky part is coming up with a proper functional test. Run.onLoad does not call any listeners. I guess you could attach a RunAction to a build when created then call purgeCache on the run map and check that Job.getLastFailedBuild does not load any builds other than the last failed one.

            Show
            jglick Jesse Glick added a comment - If my hypothesis is correct, the fix should be easy, but the tricky part is coming up with a proper functional test. Run.onLoad does not call any listeners. I guess you could attach a RunAction to a build when created then call purgeCache on the run map and check that Job.getLastFailedBuild does not load any builds other than the last failed one.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            changelog.html
            core/src/main/java/hudson/model/Job.java
            test/src/test/java/hudson/model/JobTest.java
            http://jenkins-ci.org/commit/jenkins/be5cd4fd169005f50eea8ccefac4b91a9b4c503c
            Log:
            [FIXED JENKINS-16023] Use permalinks for as many Job.getLast*Build methods as possible.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: changelog.html core/src/main/java/hudson/model/Job.java test/src/test/java/hudson/model/JobTest.java http://jenkins-ci.org/commit/jenkins/be5cd4fd169005f50eea8ccefac4b91a9b4c503c Log: [FIXED JENKINS-16023] Use permalinks for as many Job.getLast*Build methods as possible.
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #2563
            [FIXED JENKINS-16023] Use permalinks for as many Job.getLast*Build methods as possible. (Revision be5cd4fd169005f50eea8ccefac4b91a9b4c503c)

            Result = SUCCESS
            Jesse Glick : be5cd4fd169005f50eea8ccefac4b91a9b4c503c
            Files :

            • test/src/test/java/hudson/model/JobTest.java
            • core/src/main/java/hudson/model/Job.java
            • changelog.html
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #2563 [FIXED JENKINS-16023] Use permalinks for as many Job.getLast*Build methods as possible. (Revision be5cd4fd169005f50eea8ccefac4b91a9b4c503c) Result = SUCCESS Jesse Glick : be5cd4fd169005f50eea8ccefac4b91a9b4c503c Files : test/src/test/java/hudson/model/JobTest.java core/src/main/java/hudson/model/Job.java changelog.html
            Hide
            pancake pancake added a comment -

            Johno, could you please share information on how to get "Local Variable"s included in a stack trace?

            Show
            pancake pancake added a comment - Johno , could you please share information on how to get "Local Variable"s included in a stack trace?
            Hide
            johno Johno Crawford added a comment -

            @pancake VisualVM 1.3+ shows local variables in thread dumps.

            Show
            johno Johno Crawford added a comment - @pancake VisualVM 1.3+ shows local variables in thread dumps.
            Hide
            pancake pancake added a comment -

            @johno oh, indeed "Local Variable"s are shown in "Threads at the heap dump". Thanks!

            Show
            pancake pancake added a comment - @johno oh, indeed "Local Variable"s are shown in "Threads at the heap dump". Thanks!

              People

              • Assignee:
                kohsuke Kohsuke Kawaguchi
                Reporter:
                johno Johno Crawford
              • Votes:
                3 Vote for this issue
                Watchers:
                16 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: