Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49347

BlueOcean unusably slow, maybe related to having a lot of builds stored in history

    XMLWordPrintable

    Details

    • Epic Link:
    • Sprint:
      Blue Ocean 1.5 - beta 3
    • Similar Issues:

      Description

      During FOSDEM discussion about a problem we had with BlueOcean being prohibitively slow, a reasonable guess from Mark Waite was that our system in question stores a long history of builds - tens of thousands of those. Indeed, when the browser sends a request for BlueOcean (main panel, particular project, a build) the server spends a lot of time (minutes) in "wait" state for one of the CPU cores while others are idle or in small user time spending.

      Rendering the list of recent 50-100 builds or details of a particular build should not require inspecting all histories of all builds, right?

        Attachments

          Issue Links

            Activity

            Hide
            vivek Vivek Pandey added a comment -

            There were few underlying issues. One being related to how favorites were rendered (affects logged in user with long build history) and the other had to do with unnecessary and expensive data returned as part of latestRun field.

            Fix is in review stage and should be merged soon. See https://github.com/jenkinsci/blueocean-plugin/pull/1638 and  https://github.com/jenkinsci/blueocean-plugin/pull/1635.

            Show
            vivek Vivek Pandey added a comment - There were few underlying issues. One being related to how favorites were rendered (affects logged in user with long build history) and the other had to do with unnecessary and expensive data returned as part of latestRun field. Fix is in review stage and should be merged soon. See https://github.com/jenkinsci/blueocean-plugin/pull/1638  and   https://github.com/jenkinsci/blueocean-plugin/pull/1635 .
            Hide
            jamesdumay James Dumay added a comment -

            Jim Klimov Ill close this against the issues quoted by Vivek Pandey (likely to be the cause).

            However, would be great to capture a HAR file of specific page loads and capture a support bundle to help us diagnose these problems. We know there are optimizations to be make but we need specific info

            Show
            jamesdumay James Dumay added a comment - Jim Klimov Ill close this against the issues quoted by Vivek Pandey (likely to be the cause). However, would be great to capture a HAR file of specific page loads and capture a support bundle to help us diagnose these problems. We know there are optimizations to be make but we need specific info
            Hide
            jimklimov Jim Klimov added a comment - - edited

            Added HAR and respective screenshots (console is rather empty and uninteresting, but load times are spectacular).

            UPDATE: Added somewhat redacted support data.

            Statistics-wise, at the moment there are about 40k builds stored in history (and yes, much of the time spent in rendering BO can be Java walking the filesystem to find and parse all that build data):

            [root@jenkins2 jenkins/jobs]# sync; echo 3 > /proc/sys/vm/drop_caches
            [root@jenkins2 jenkins/jobs]# find . -name build.xml | wc -l
            39847
            
            real    12m40.689s
            user    0m12.259s
            sys     0m36.710s
            

            and 94 known "users" (many of which were auto-registered from upstream/3rd party projects' commits that were built here; about 20 actual internal users). Listing $JENKINS_URL/view/Masters/asynchPeople/ or going into the active users and listing builds they are associated with at $JENKINS_URL/user/myusername/builds also takes several minutes, at least first time after a filesystem cache flush for cleaner results (and comparably long with a "dirty" cache too).

            The system is a KVM VM, has 4Gb RAM assigned and about 1.3Gb of that is free. I'll try bumping it to 8Gb so more FS (meta)data can be cached, at least...

            Show
            jimklimov Jim Klimov added a comment - - edited Added HAR and respective screenshots (console is rather empty and uninteresting, but load times are spectacular). UPDATE: Added somewhat redacted support data. Statistics-wise, at the moment there are about 40k builds stored in history (and yes, much of the time spent in rendering BO can be Java walking the filesystem to find and parse all that build data): [root@jenkins2 jenkins/jobs]# sync; echo 3 > /proc/sys/vm/drop_caches [root@jenkins2 jenkins/jobs]# find . -name build.xml | wc -l 39847 real 12m40.689s user 0m12.259s sys 0m36.710s and 94 known "users" (many of which were auto-registered from upstream/3rd party projects' commits that were built here; about 20 actual internal users). Listing $JENKINS_URL/view/Masters/asynchPeople/ or going into the active users and listing builds they are associated with at $JENKINS_URL/user/myusername/builds also takes several minutes, at least first time after a filesystem cache flush for cleaner results (and comparably long with a "dirty" cache too). The system is a KVM VM, has 4Gb RAM assigned and about 1.3Gb of that is free. I'll try bumping it to 8Gb so more FS (meta)data can be cached, at least...
            Hide
            jamesdumay James Dumay added a comment -

            Vivek Pandey - Jim Klimov has provided some HARs and a support bundle, so it should be possible to see the problem he is experiencing now.

            Show
            jamesdumay James Dumay added a comment - Vivek Pandey - Jim Klimov has provided some HARs and a support bundle, so it should be possible to see the problem he is experiencing now.
            Hide
            vivek Vivek Pandey added a comment -

            I looked at the attached HAR file. Most slowness are around how favorites are pre-fetched and loaded subsequently.

            Fixes for optimized favorites loading (https://github.com/jenkinsci/blueocean-plugin/pull/1638) and minimizing returns in latestRun in pipeline should fixed this issue (https://github.com/jenkinsci/blueocean-plugin/pull/1635).

            These fixes went in 1.5.0-beta-1. I recommend trying latest 1.5.0-beta-2, and report back/reopen if the problem persists.

            Show
            vivek Vivek Pandey added a comment - I looked at the attached HAR file. Most slowness are around how favorites are pre-fetched and loaded subsequently. Fixes for optimized favorites loading ( https://github.com/jenkinsci/blueocean-plugin/pull/1638 ) and minimizing returns in latestRun in pipeline should fixed this issue ( https://github.com/jenkinsci/blueocean-plugin/pull/1635 ). These fixes went in 1.5.0-beta-1. I recommend trying latest 1.5.0-beta-2, and report back/reopen if the problem persists.

              People

              • Assignee:
                vivek Vivek Pandey
                Reporter:
                jimklimov Jim Klimov
              • Votes:
                1 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: