Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43208

Very slow dashboard and /search response in some cases (slowing dashboard load)

    XMLWordPrintable

    Details

    • Epic Link:
    • Sprint:
      Blue Ocean 1.0-rc4
    • Similar Issues:

      Description

      User has reported a performance problem on 1.0-rc3 loading the dashboard.

      It appears from a brief analysis of their HAR file – attached here – that /blue/pipelines takes 33 seconds to load and search taking 30s or so: 

      JENKINSDOMAIN/blue/rest/search/?q=type:pipeline;excludedFromFlattening:jenkins.branch.MultiBranchProject,hudson.matrix.MatrixProject&filter=no-folders&start=0&limit=26. The response is about the normal size expected (perhaps slightly larger, but not large enough to explain the load time, content can be seen in the har file).

       

       

       

        Attachments

          Issue Links

            Activity

            jamesdumay James Dumay created issue -
            jamesdumay James Dumay made changes -
            Field Original Value New Value
            Epic Link JENKINS-35759 [ 171771 ]
            jamesdumay James Dumay made changes -
            Rank Ranked higher
            michaelneale Michael Neale made changes -
            Description User has [reported a performance problem on 1.0-rc2|https://groups.google.com/forum/#!topic/jenkinsci-dev/zCz4Zxye_uo] loading the dashboard.

            It appears from a brief analysis of their HAR file – [attached here|https://groups.google.com/forum/#!topic/jenkinsci-dev/zCz4Zxye_uo] – that /blue/pipelines takes 33 seconds to load.

            User has [reported a performance problem on 1.0-rc|https://groups.google.com/forum/#!topic/jenkinsci-dev/zCz4Zxye_uo]3 loading the dashboard.

            It appears from a brief analysis of their HAR file – [attached here|https://groups.google.com/forum/#!topic/jenkinsci-dev/zCz4Zxye_uo] – that /blue/pipelines takes 33 seconds to load - this is due to the search taking pretty much all that time: 

             

            [http://usalsd-build01:8080/blue/rest/search/?q=type:pipeline;excludedFromFlattening:jenkins.branch.MultiBranchProject,hudson.matrix.MatrixProject&filter=no-folders&start=0&limit=26]

            is the request that takes all the time. 
            Hide
            michaelneale Michael Neale added a comment - - edited

            cc Marley Kudiabor - for following up, some questions for you: 

            • Your non slow jenkins server - if you update that to rc3 - does it then slow down? 
            • If you downgrade the slow server to an older release, does it speed up?  (if so, which version?)
            • If you load the slow page again, does it take 60s or 1 minute or less for subsequent loads? 
            • Is the "classic" dashboard slow or other pages slow in any way on the slow server?

            Just trying to narrow down if it is due to plugins/environment for the slow jenkins, of if it is something introduced in a blueocean plugin. Unfortunately there may not be enough information to go on in the har file (there may be some URLs we can try if Vivek Pandey suggests anything). This is a very small number of jobs and many people use it with 100s or 1000s, so this is quite odd. 

             

            Further more, you could use the https://wiki.jenkins-ci.org/display/JENKINS/Support+Core+Plugin plugin to provide extra data as to what is running and what are slow requests etc, if this isn't enough, but the above questions would be most helpful to answer. 

            Show
            michaelneale Michael Neale added a comment - - edited cc Marley Kudiabor - for following up, some questions for you:  Your non slow jenkins server - if you update that to rc3 - does it then slow down?  If you downgrade the slow server to an older release, does it speed up?  (if so, which version?) If you load the slow page again, does it take 60s or 1 minute or less for subsequent loads?  Is the "classic" dashboard slow or other pages slow in any way on the slow server? Just trying to narrow down if it is due to plugins/environment for the slow jenkins, of if it is something introduced in a blueocean plugin. Unfortunately there may not be enough information to go on in the har file (there may be some URLs we can try if Vivek Pandey suggests anything). This is a very small number of jobs and many people use it with 100s or 1000s, so this is quite odd.    Further more, you could use the https://wiki.jenkins-ci.org/display/JENKINS/Support+Core+Plugin  plugin to provide extra data as to what is running and what are slow requests etc, if this isn't enough, but the above questions would be most helpful to answer. 
            michaelneale Michael Neale made changes -
            Summary Performance issue loading dasbhoard Very slow /search response in some cases (slowing dashboard load)
            michaelneale Michael Neale made changes -
            Description User has [reported a performance problem on 1.0-rc|https://groups.google.com/forum/#!topic/jenkinsci-dev/zCz4Zxye_uo]3 loading the dashboard.

            It appears from a brief analysis of their HAR file – [attached here|https://groups.google.com/forum/#!topic/jenkinsci-dev/zCz4Zxye_uo] – that /blue/pipelines takes 33 seconds to load - this is due to the search taking pretty much all that time: 

             

            [http://usalsd-build01:8080/blue/rest/search/?q=type:pipeline;excludedFromFlattening:jenkins.branch.MultiBranchProject,hudson.matrix.MatrixProject&filter=no-folders&start=0&limit=26]

            is the request that takes all the time. 
            User has [reported a performance problem on 1.0-rc|https://groups.google.com/forum/#!topic/jenkinsci-dev/zCz4Zxye_uo]3 loading the dashboard.

            It appears from a brief analysis of their HAR file – [attached here|https://groups.google.com/forum/#!topic/jenkinsci-dev/zCz4Zxye_uo] – that /blue/pipelines takes 33 seconds to load - this is due to the search taking pretty much all that time: 

            [http://usalsd-build01:8080/blue/rest/search/?q=type:pipeline;excludedFromFlattening:jenkins.branch.MultiBranchProject,hudson.matrix.MatrixProject&filter=no-folders&start=0&limit=26]

            is the request that takes all the time. 
            michaelneale Michael Neale made changes -
            Description User has [reported a performance problem on 1.0-rc|https://groups.google.com/forum/#!topic/jenkinsci-dev/zCz4Zxye_uo]3 loading the dashboard.

            It appears from a brief analysis of their HAR file – [attached here|https://groups.google.com/forum/#!topic/jenkinsci-dev/zCz4Zxye_uo] – that /blue/pipelines takes 33 seconds to load - this is due to the search taking pretty much all that time: 

            [http://usalsd-build01:8080/blue/rest/search/?q=type:pipeline;excludedFromFlattening:jenkins.branch.MultiBranchProject,hudson.matrix.MatrixProject&filter=no-folders&start=0&limit=26]

            is the request that takes all the time. 
            User has [reported a performance problem on 1.0-rc|https://groups.google.com/forum/#!topic/jenkinsci-dev/zCz4Zxye_uo]3 loading the dashboard.

            It appears from a brief analysis of their HAR file – [attached here|https://groups.google.com/forum/#!topic/jenkinsci-dev/zCz4Zxye_uo] – that /blue/pipelines takes 33 seconds to load - this is due to the search taking pretty much all that time: 

            [http://usalsd-build01:8080/blue/rest/search/?q=type:pipeline;excludedFromFlattening:jenkins.branch.MultiBranchProject,hudson.matrix.MatrixProject&filter=no-folders&start=0&limit=26]. The response is about the normal size expected (perhaps slightly larger, but not large enough to explain the load time). 

            is the request that takes all the time (everything else seems normal). 

             

             
            michaelneale Michael Neale made changes -
            Attachment hardyhar.zip [ 36821 ]
            Hide
            michaelneale Michael Neale added a comment -

            Vivek Pandey would you suggest any experiments on that search query that may hint at what is causing the slow down? 

            Show
            michaelneale Michael Neale added a comment - Vivek Pandey would you suggest any experiments on that search query that may hint at what is causing the slow down? 
            michaelneale Michael Neale made changes -
            Description User has [reported a performance problem on 1.0-rc|https://groups.google.com/forum/#!topic/jenkinsci-dev/zCz4Zxye_uo]3 loading the dashboard.

            It appears from a brief analysis of their HAR file – [attached here|https://groups.google.com/forum/#!topic/jenkinsci-dev/zCz4Zxye_uo] – that /blue/pipelines takes 33 seconds to load - this is due to the search taking pretty much all that time: 

            [http://usalsd-build01:8080/blue/rest/search/?q=type:pipeline;excludedFromFlattening:jenkins.branch.MultiBranchProject,hudson.matrix.MatrixProject&filter=no-folders&start=0&limit=26]. The response is about the normal size expected (perhaps slightly larger, but not large enough to explain the load time). 

            is the request that takes all the time (everything else seems normal). 

             

             
            User has [reported a performance problem on 1.0-rc|https://groups.google.com/forum/#!topic/jenkinsci-dev/zCz4Zxye_uo]3 loading the dashboard.

            It appears from a brief analysis of their HAR file – [attached here|https://groups.google.com/forum/#!topic/jenkinsci-dev/zCz4Zxye_uo] – that /blue/pipelines takes 33 seconds to load - this is due to the search taking pretty much all that time: 

            [http://usalsd-build01:8080/blue/rest/search/?q=type:pipeline;excludedFromFlattening:jenkins.branch.MultiBranchProject,hudson.matrix.MatrixProject&filter=no-folders&start=0&limit=26]. The response is about the normal size expected (perhaps slightly larger, but not large enough to explain the load time, content can be seen in the har file).

            is the request that takes all the time (everything else seems normal). 

             

             
            michaelneale Michael Neale made changes -
            Summary Very slow /search response in some cases (slowing dashboard load) Very slow dashboard and /search response in some cases (slowing dashboard load)
            michaelneale Michael Neale made changes -
            Description User has [reported a performance problem on 1.0-rc|https://groups.google.com/forum/#!topic/jenkinsci-dev/zCz4Zxye_uo]3 loading the dashboard.

            It appears from a brief analysis of their HAR file – [attached here|https://groups.google.com/forum/#!topic/jenkinsci-dev/zCz4Zxye_uo] – that /blue/pipelines takes 33 seconds to load - this is due to the search taking pretty much all that time: 

            [http://usalsd-build01:8080/blue/rest/search/?q=type:pipeline;excludedFromFlattening:jenkins.branch.MultiBranchProject,hudson.matrix.MatrixProject&filter=no-folders&start=0&limit=26]. The response is about the normal size expected (perhaps slightly larger, but not large enough to explain the load time, content can be seen in the har file).

            is the request that takes all the time (everything else seems normal). 

             

             
            User has [reported a performance problem on 1.0-rc|https://groups.google.com/forum/#!topic/jenkinsci-dev/zCz4Zxye_uo]3 loading the dashboard.

            It appears from a brief analysis of their HAR file – [attached here|https://groups.google.com/forum/#!topic/jenkinsci-dev/zCz4Zxye_uo] – that /blue/pipelines takes 33 seconds to load and search taking 30s or so: 

            [http://usalsd-build01:8080/blue/rest/search/?q=type:pipeline;excludedFromFlattening:jenkins.branch.MultiBranchProject,hudson.matrix.MatrixProject&filter=no-folders&start=0&limit=26]. The response is about the normal size expected (perhaps slightly larger, but not large enough to explain the load time, content can be seen in the har file).

            is the request that takes all the time (everything else seems normal). 

             

             
            michaelneale Michael Neale made changes -
            Description User has [reported a performance problem on 1.0-rc|https://groups.google.com/forum/#!topic/jenkinsci-dev/zCz4Zxye_uo]3 loading the dashboard.

            It appears from a brief analysis of their HAR file – [attached here|https://groups.google.com/forum/#!topic/jenkinsci-dev/zCz4Zxye_uo] – that /blue/pipelines takes 33 seconds to load and search taking 30s or so: 

            [http://usalsd-build01:8080/blue/rest/search/?q=type:pipeline;excludedFromFlattening:jenkins.branch.MultiBranchProject,hudson.matrix.MatrixProject&filter=no-folders&start=0&limit=26]. The response is about the normal size expected (perhaps slightly larger, but not large enough to explain the load time, content can be seen in the har file).

            is the request that takes all the time (everything else seems normal). 

             

             
            User has [reported a performance problem on 1.0-rc|https://groups.google.com/forum/#!topic/jenkinsci-dev/zCz4Zxye_uo]3 loading the dashboard.

            It appears from a brief analysis of their HAR file – [attached here|https://groups.google.com/forum/#!topic/jenkinsci-dev/zCz4Zxye_uo] – that /blue/pipelines takes 33 seconds to load and search taking 30s or so: 

            [http://usalsd-build01:8080/blue/rest/search/?q=type:pipeline;excludedFromFlattening:jenkins.branch.MultiBranchProject,hudson.matrix.MatrixProject&filter=no-folders&start=0&limit=26]. The response is about the normal size expected (perhaps slightly larger, but not large enough to explain the load time, content can be seen in the har file).

             

             

             
            michaelneale Michael Neale made changes -
            Attachment hardyhar.zip [ 36821 ]
            Hide
            kudiaborm Marley Kudiabor added a comment - - edited

            Unfortunately today isn't the best day to update the jenkins server itself. Blue Ocean was updated to rc3 though and it was still slow before and after. I believe previously it was on b23 or so?

            The slow page takes about the same amount of time to load on first load and sequential loads with or without caching.

            The classic dashboard is still very snappy.

            Attached is a log file from after a long load. 
            Please email me directly for the password. I've emailed those already mentioned in this ticket the file and password pre-emptively

            Show
            kudiaborm Marley Kudiabor added a comment - - edited Unfortunately today isn't the best day to update the jenkins server itself. Blue Ocean was updated to rc3 though and it was still slow before and after. I believe previously it was on b23 or so? The slow page takes about the same amount of time to load on first load and sequential loads with or without caching. The classic dashboard is still very snappy. Attached is a log file from after a long load.  Please email me directly for the password. I've emailed those already mentioned in this ticket the file and password pre-emptively
            kudiaborm Marley Kudiabor made changes -
            Attachment support_2017-03-30_14.22.52.zip [ 36824 ]
            Attachment support_2017-03-30_14.23.07.zip [ 36825 ]
            Attachment support_2017-03-30_14.23.17.zip [ 36826 ]
            Attachment support_2017-03-30_14.23.36.zip [ 36827 ]
            Attachment support_2017-03-30_14.24.50.zip [ 36828 ]
            kudiaborm Marley Kudiabor made changes -
            Attachment support_2017-03-30_14.22.52.zip [ 36824 ]
            kudiaborm Marley Kudiabor made changes -
            Attachment support_2017-03-30_14.23.07.zip [ 36825 ]
            kudiaborm Marley Kudiabor made changes -
            Attachment support_2017-03-30_14.23.17.zip [ 36826 ]
            kudiaborm Marley Kudiabor made changes -
            Attachment support_2017-03-30_14.23.36.zip [ 36827 ]
            kudiaborm Marley Kudiabor made changes -
            Attachment support_2017-03-30_14.24.50.zip [ 36828 ]
            kudiaborm Marley Kudiabor made changes -
            Hide
            vivek Vivek Pandey added a comment - - edited

            On Analyzing HAR file, looks like Job config history plugin actions are causing this slowdown. Brief look at this plugin code tells the history is not cached so each history item results in to read from file system. I could reproduce it locally where just couple of freestyle jobs with config history of in 30 or 40 resulted in almost 8 to 9 times degradation.

             

            Marley Kudiabor is it possible for you to set max number of config history visible per page to lets say 5 or so to see if it makes difference. At least it will validate the theory. You can find it at  Manage Jenkins->System Configuration -> Job Config History -> Advanced -> Max number of history entries to show per page.

            At present going over favorited items (26 of them) have config history items all the way from 3 to 143. 

            Here are the size of job config history items per pipeline.

            [17, 3, 7, 3, 11, 11, 4, 30, 143, 14, 4, 7, 3, 8, 4, 12, 13, 24, 4, 20, 4, 11, 9, 53, 17, 11]

            On blueocean side of things: Actions could bring such performance issues as blueocean currently is fetching them eagerly, which is bad. To fix it we need to have UI provide filter using tree api of search (/blue/rest/search API). Prefetch is different, here it's server side generated HTML with generated favorite list, it needs to be fixed so that it processes UI provided tree parameters in /blue/pipeline/ HTTP call to generate list of favorites.

            Show
            vivek Vivek Pandey added a comment - - edited On Analyzing HAR file, looks like Job config history plugin actions are causing this slowdown. Brief look at this plugin code tells the history is not cached so each history item results in to read from file system. I could reproduce it locally where just couple of freestyle jobs with config history of in 30 or 40 resulted in almost 8 to 9 times degradation.   Marley Kudiabor is it possible for you to set max number of config history visible per page to lets say 5 or so to see if it makes difference. At least it will validate the theory. You can find it at  Manage Jenkins->System Configuration -> Job Config History -> Advanced -> Max number of history entries to show per page. At present going over favorited items (26 of them) have config history items all the way from 3 to 143.  Here are the size of job config history items per pipeline. [17, 3, 7, 3, 11, 11, 4, 30, 143, 14, 4, 7, 3, 8, 4, 12, 13, 24, 4, 20, 4, 11, 9, 53, 17, 11] On blueocean side of things: Actions could bring such performance issues as blueocean currently is fetching them eagerly, which is bad. To fix it we need to have UI provide filter using tree api of search (/blue/rest/search API). Prefetch is different, here it's server side generated HTML with generated favorite list, it needs to be fixed so that it processes UI provided tree parameters in /blue/pipeline/ HTTP call to generate list of favorites.
            Hide
            kudiaborm Marley Kudiabor added a comment -

            Unfortunately neither decreasing history objects or even completely removing the config-history folder from %JENKINSHOME% affect the lag. Perhaps the problem is somewhere else in my configuration.

            I'll be off work until Monday so I apologize for any delay in response until then
             

            Show
            kudiaborm Marley Kudiabor added a comment - Unfortunately neither decreasing history objects or even completely removing the config-history folder from %JENKINSHOME% affect the lag. Perhaps the problem is somewhere else in my configuration. I'll be off work until Monday so I apologize for any delay in response until then  
            Hide
            michaelneale Michael Neale added a comment -

            The job config history plugin is infamous for introducing problems in various places (unless you use ramdisk, rarely save changes to config), so this isn't too surprising. 

            If you dont' want to remove it (you probably have it for a reason), then there will be a fix coming, and blue ocean shouldn't be slowed down by this either. 

            The fix in summary: 

            • don't just fetch everything about a pipeline just in case it is needed 
            • don't return history data that isn't wanted anyway

            One of these should land soon (one is a longer term, one is a shorter term enhancement).

             

            Thanks! 

             

            Show
            michaelneale Michael Neale added a comment - The job config history plugin is infamous for introducing problems in various places (unless you use ramdisk, rarely save changes to config), so this isn't too surprising.  If you dont' want to remove it (you probably have it for a reason), then there will be a fix coming, and blue ocean shouldn't be slowed down by this either.  The fix in summary:  don't just fetch everything about a pipeline just in case it is needed  don't return history data that isn't wanted anyway One of these should land soon (one is a longer term, one is a shorter term enhancement).   Thanks!   
            vivek Vivek Pandey made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            vivek Vivek Pandey made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            jamesdumay James Dumay made changes -
            Status In Review [ 10005 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            michaelneale Michael Neale made changes -
            Remote Link This issue links to "Wiki Page (Jenkins Wiki)" [ 15905 ]
            kudiaborm Marley Kudiabor made changes -
            Description User has [reported a performance problem on 1.0-rc|https://groups.google.com/forum/#!topic/jenkinsci-dev/zCz4Zxye_uo]3 loading the dashboard.

            It appears from a brief analysis of their HAR file – [attached here|https://groups.google.com/forum/#!topic/jenkinsci-dev/zCz4Zxye_uo] – that /blue/pipelines takes 33 seconds to load and search taking 30s or so: 

            [http://usalsd-build01:8080/blue/rest/search/?q=type:pipeline;excludedFromFlattening:jenkins.branch.MultiBranchProject,hudson.matrix.MatrixProject&filter=no-folders&start=0&limit=26]. The response is about the normal size expected (perhaps slightly larger, but not large enough to explain the load time, content can be seen in the har file).

             

             

             
            User has [reported a performance problem on 1.0-rc|https://groups.google.com/forum/#!topic/jenkinsci-dev/zCz4Zxye_uo]3 loading the dashboard.

            It appears from a brief analysis of their HAR file – [attached here|https://groups.google.com/forum/#!topic/jenkinsci-dev/zCz4Zxye_uo] – that /blue/pipelines takes 33 seconds to load and search taking 30s or so: 

            [JENKINSDOMAIN/blue/rest/search/?q=type:pipeline;excludedFromFlattening:jenkins.branch.MultiBranchProject,hudson.matrix.MatrixProject&filter=no-folders&start=0&limit=26|http://usalsd-build01:8080/blue/rest/search/?q=type:pipeline;excludedFromFlattening:jenkins.branch.MultiBranchProject,hudson.matrix.MatrixProject&filter=no-folders&start=0&limit=26]. The response is about the normal size expected (perhaps slightly larger, but not large enough to explain the load time, content can be seen in the har file).

             

             

             
            Hide
            kudiaborm Marley Kudiabor added a comment - - edited

            Modified path in description to remove our jenkins domain from google search results

            Show
            kudiaborm Marley Kudiabor added a comment - - edited Modified path in description to remove our jenkins domain from google search results
            Hide
            schulzha Hans Schulz added a comment - - edited

            Repoening because the issue is not fixed.

            As Marley Kudiabor wrote, the config history plugin did not cause the long loading times.

            I am experiencing the exact same issue, opening blue ocean takes 18 seconds for us on a very fast server. Deleting the config history did not help.

            I've noticed that the dashboard loads quickly when I'm not authenticated.

            Jenkins version 2.46.3, BlueOcean 1.1.2

            Show
            schulzha Hans Schulz added a comment - - edited Repoening because the issue is not fixed. As Marley Kudiabor wrote, the config history plugin did not cause the long loading times. I am experiencing the exact same issue, opening blue ocean takes 18 seconds for us on a very fast server. Deleting the config history did not help. I've noticed that the dashboard loads quickly when I'm not authenticated. Jenkins version 2.46.3, BlueOcean 1.1.2
            schulzha Hans Schulz made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            Hide
            michaelneale Michael Neale added a comment -

            Hans Schulz it would be better to open a fresh ticket than re-open this one with all the noise as this isn't related to config history plugin (likey another plugin, or number of favourites, could be a lot of things).

            Then can chase it down. 

            Show
            michaelneale Michael Neale added a comment - Hans Schulz it would be better to open a fresh ticket than re-open this one with all the noise as this isn't related to config history plugin (likey another plugin, or number of favourites, could be a lot of things). Then can chase it down. 
            Hide
            jamesdumay James Dumay added a comment - - edited

            Hans Schulz please open a new issue and email me a HAR file for the page load at jdumay@cloudbees.com

            If you are a CloudBees customer you will get a better turn around by filing a ticket at support.cloudbees.com and attaching a the HAR file there.

            Show
            jamesdumay James Dumay added a comment - - edited Hans Schulz please open a new issue and email me a HAR file for the page load at jdumay@cloudbees.com If you are a CloudBees customer you will get a better turn around by filing a ticket at support.cloudbees.com and attaching a the HAR file there.
            jamesdumay James Dumay made changes -
            Status Reopened [ 4 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            jbriden Jenn Briden made changes -
            Status Resolved [ 5 ] Closed [ 6 ]

              People

              • Assignee:
                vivek Vivek Pandey
                Reporter:
                jamesdumay James Dumay
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: