Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44995

Very slow activity/pipeline screen load (often when logged in)

    Details

    • Similar Issues:
    • Epic Link:
    • Sprint:
      Blue Ocean 1.2-beta1, Blue Ocean 1.2-beta2, Blue Ocean 1.4 - beta 3, Blue Ocean 1.5 - beta 1

      Description

      (to investigate)

       

      • Some people are finding dashboard/pipeline screens slow when logged in vs not logged in (see appropriate comments below) 
        • See comments and support bundles below for this - eg via Brian Saville. In some cases it is related to number of runs. 
      • Some users see activity screen as very slow when there are a large number of runs (this seems to be the more common and serious case...)

       

       

       

      — ORIGINAL TICKET —

      I've noticed that the dashboard loads quickly when I'm not authenticated.

      (classic loads normally)

       

      Deleting the config history like suggested in https://issues.jenkins-ci.org/browse/JENKINS-43208 did not work.

      I have sent an HAR file via email to James Dumay.

      Jenkins version 2.46.3, BlueOcean 1.1.2

        Attachments

        1. haranalysis.png
          haranalysis.png
          118 kB
        2. plugins.txt
          9 kB
        3. support_2017-11-17_08.03.54.zip
          743 kB
        4. support_2017-11-17_08.03.56.zip
          743 kB
        5. support_2017-11-17_08.03.59.zip
          743 kB
        6. support_2017-11-17_08.04.02.zip
          743 kB
        7. support_2017-11-17_08.04.13.zip
          743 kB
        8. support_2017-11-17_08.04.18.zip
          742 kB
        9. support_2017-11-17_11.26.08.zip
          401 kB
        10. support_2017-11-20_23.21.08.zip
          269 kB

          Issue Links

            Activity

            Hide
            jamesdumay James Dumay added a comment - - edited

            Hans Schulz what authentication backend are you using? LDAP? What plugin?

            I received the HAR file - thank you

            Show
            jamesdumay James Dumay added a comment - - edited Hans Schulz what authentication backend are you using? LDAP? What plugin? I received the HAR file - thank you
            Hide
            jamesdumay James Dumay added a comment -

            Hans Schulz it looks like most requests to the REST API for things that could be slow (listing pipelines, fetching favourites) are returning within a few milliseconds. The page load for Blue Ocean does not.

            I have a few questions:

            • Is Blue Ocean behind a HTTP proxy?
            • Can you please generate a support bundle and email it to me. This will give me some indication of what plugins are running on your system etc
            Show
            jamesdumay James Dumay added a comment - Hans Schulz it looks like most requests to the REST API for things that could be slow (listing pipelines, fetching favourites) are returning within a few milliseconds. The page load for Blue Ocean does not. I have a few questions: Is Blue Ocean behind a HTTP proxy? Can you please generate a support bundle and email it to me. This will give me some indication of what plugins are running on your system etc
            Hide
            schulzha Hans Schulz added a comment -

            I am using Github Authorization backend with a Github Enterprise instance.

            Jenkins is behind an Apache2 reverse-proxy for TLS, configured like described here: https://wiki.jenkins.io/display/JENKINS/Running+Jenkins+behind+Apache.

            Apart from that, our corporate network uses a proxy to connect to the internet, but that shouldn't be involved in loading blue ocean.

            I have sent you the support bundle.

            Show
            schulzha Hans Schulz added a comment - I am using Github Authorization backend with a Github Enterprise instance. Jenkins is behind an Apache2 reverse-proxy for TLS, configured like described here: https://wiki.jenkins.io/display/JENKINS/Running+Jenkins+behind+Apache . Apart from that, our corporate network uses a proxy to connect to the internet, but that shouldn't be involved in loading blue ocean. I have sent you the support bundle.
            Hide
            jamesdumay James Dumay added a comment -

            Thanks!

            Another thing: would you be able to perform a thread dump of the server while the dashboard is loading slowly? That could give us some extra ideas of what's happening. Just a few thread dumps every 5 seconds as it's loading would be perfect

            Show
            jamesdumay James Dumay added a comment - Thanks! Another thing: would you be able to perform a thread dump of the server while the dashboard is loading slowly? That could give us some extra ideas of what's happening. Just a few thread dumps every 5 seconds as it's loading would be perfect
            Hide
            schulzha Hans Schulz added a comment -

            I have sent you the thread dumps.

            Show
            schulzha Hans Schulz added a comment - I have sent you the thread dumps.
            Hide
            bksaville Brian Saville added a comment -

            I'm running into the same issue here. GitHub Auth backend, same as Hans.

            Show
            bksaville Brian Saville added a comment - I'm running into the same issue here. GitHub Auth backend, same as Hans.
            Hide
            jamesdumay James Dumay added a comment -

            Hans Schulz I see you are using github-oauth 0.27 which is the same version we use on [BlueOcean.io|ci.blueocean.io] without any troubles. Hmm, could you try accessing without the proxy in front? I just want to be 100% sure that its not interfering with requests.

            Brian Saville are you using the github-oauth 0.27 too? Can you provide a list of your installed plugins attached as a text file to this issue please?

            Show
            jamesdumay James Dumay added a comment - Hans Schulz I see you are using github-oauth 0.27 which is the same version we use on [BlueOcean.io|ci.blueocean.io] without any troubles. Hmm, could you try accessing without the proxy in front? I just want to be 100% sure that its not interfering with requests. Brian Saville are you using the github-oauth 0.27 too? Can you provide a list of your installed plugins attached as a text file to this issue please?
            Hide
            bksaville Brian Saville added a comment - - edited

            Attached my list of installed plugins - we have a couple that our internal only, but the majority are on latest or close to latest versions. And yes, we are using version 0.27 as well. I should note that even though it reports blue ocean 1.1.2 as installed, I believe it hasn't been activated yet since I haven't restarted since upgrading the blue ocean plugins, so we're still on 1.1.1 as far as I know.

            Show
            bksaville Brian Saville added a comment - - edited Attached my list of installed plugins - we have a couple that our internal only, but the majority are on latest or close to latest versions. And yes, we are using version 0.27 as well. I should note that even though it reports blue ocean 1.1.2 as installed, I believe it hasn't been activated yet since I haven't restarted since upgrading the blue ocean plugins, so we're still on 1.1.1 as far as I know.
            Hide
            jamesdumay James Dumay added a comment -

            Brian Saville OK, can you please report back after you restart?

            Show
            jamesdumay James Dumay added a comment - Brian Saville OK, can you please report back after you restart?
            Hide
            schulzha Hans Schulz added a comment -

            James Dumay I actually cannot try without the proxy, since it does our TLS configuration and OAuth does not work without HTTPS. But it is just a plain Apache2 configured as the wiki describes, so it should not be a problem.

            Show
            schulzha Hans Schulz added a comment - James Dumay I actually cannot try without the proxy, since it does our TLS configuration and OAuth does not work without HTTPS. But it is just a plain Apache2 configured as the wiki describes, so it should not be a problem.
            Hide
            jamesdumay James Dumay added a comment -

            Hi Hans Schulz,

            I couldn't find the heap dump in your log files.

            Could you try the following:

            1. Find the process id (pid) of Jenkins
            2. Try to load the slow page
            3. Run on the terminal kill -3 <jenkins pid> where <jenkins pid> is the process ID. Run the command every 5 seconds as your request is being processed.
            4. Send me the server log using the secure drop link I offered previously

            Thanks,
            James

            Show
            jamesdumay James Dumay added a comment - Hi Hans Schulz , I couldn't find the heap dump in your log files. Could you try the following: Find the process id (pid) of Jenkins Try to load the slow page Run on the terminal kill -3 <jenkins pid> where <jenkins pid> is the process ID. Run the command every 5 seconds as your request is being processed. Send me the server log using the secure drop link I offered previously Thanks, James
            Hide
            bksaville Brian Saville added a comment -

            James Dumay, I'm not sure what might have changed, but I haven't noticed this problem for quite awhile now and I've been logged in most of the time. I would call this fixed for us, but really not sure what changed as I haven't updated any plugins or anything that I remember.

            Show
            bksaville Brian Saville added a comment - James Dumay , I'm not sure what might have changed, but I haven't noticed this problem for quite awhile now and I've been logged in most of the time. I would call this fixed for us, but really not sure what changed as I haven't updated any plugins or anything that I remember.
            Hide
            jamesdumay James Dumay added a comment -

            Brian Saville I suspect it was some changes we made to test reporting that is causing the slowness. The good news is that we've done some profiling, identified performance issues and pushed a fix into Blue Ocean 1.2. We have a release going out during Tuesday next week that you can try. If its still a problem, we can look into this further.

            Show
            jamesdumay James Dumay added a comment - Brian Saville I suspect it was some changes we made to test reporting that is causing the slowness. The good news is that we've done some profiling, identified performance issues and pushed a fix into Blue Ocean 1.2. We have a release going out during Tuesday next week that you can try. If its still a problem, we can look into this further.
            Hide
            bksaville Brian Saville added a comment -

            Great, thanks!

            Show
            bksaville Brian Saville added a comment - Great, thanks!
            Hide
            schulzha Hans Schulz added a comment -

            I just checked, and BlueOcean is loading fast for us, too. Thanks!

            Show
            schulzha Hans Schulz added a comment - I just checked, and BlueOcean is loading fast for us, too. Thanks!
            Hide
            schulzha Hans Schulz added a comment - - edited

            Somewhere last week, after updating plugins and Jenkins, the issue came back for us.

            Fortuantely, I think I found the root cause of this issue, or at least part of it: I noticed that the autofavorite plugin went crazy on me and I had multiple hundred favorites. After I deleted them BlueOcean loads almost as fast as unauthenticated.

             

            Show
            schulzha Hans Schulz added a comment - - edited Somewhere last week, after updating plugins and Jenkins, the issue came back for us. Fortuantely, I think I found the root cause of this issue, or at least part of it: I noticed that the autofavorite plugin went crazy on me and I had multiple hundred favorites. After I deleted them BlueOcean loads almost as fast as unauthenticated.  
            Hide
            oz123 Oz Tiram added a comment -

            I am experiencing this issue too. Using Jenkins 2.60.3 with BlueOcean 1.2.1.

             

            BlueOcen takes ages to load.

            Show
            oz123 Oz Tiram added a comment - I am experiencing this issue too. Using Jenkins 2.60.3 with BlueOcean 1.2.1.   BlueOcen takes ages to load.
            Hide
            michaelneale Michael Neale added a comment -

            A support core bundle please? 

            Show
            michaelneale Michael Neale added a comment - A support core bundle please? 
            Hide
            sbabu Sunil Babu added a comment -

            I am also experiencing same issue with Jenkins 2.78 and BlueOcean 1.2.1. My setup is also similar to Hans with apache, https, Github Authorization, Github Enterprise)

            Show
            sbabu Sunil Babu added a comment - I am also experiencing same issue with Jenkins 2.78 and BlueOcean 1.2.1. My setup is also similar to Hans with apache, https, Github Authorization, Github Enterprise)
            Hide
            michaelneale Michael Neale added a comment -

            using https://wiki.jenkins.io/display/JENKINS/Support+Core+Plugin and attaching can help diagnose, otherwise there isn't much that can be done based on the information given. 

            Show
            michaelneale Michael Neale added a comment - using https://wiki.jenkins.io/display/JENKINS/Support+Core+Plugin  and attaching can help diagnose, otherwise there isn't much that can be done based on the information given. 
            Hide
            michaelneale Michael Neale added a comment -

            Oz Tiram Sunil Babu if you could use the support core plugin above - that would help a lot, and point to what is taking all the time. 

            Show
            michaelneale Michael Neale added a comment - Oz Tiram Sunil Babu if you could use the support core plugin above - that would help a lot, and point to what is taking all the time. 
            Hide
            seckler Steffen Seckler added a comment - - edited

            I do experience the same problem. I have attached the necessary support files.

            support_2017-11-17_08.03.54.zip support_2017-11-17_08.04.02.zip

            It happens for me when accessing a pipeline job (using the non-declarative version, called MardynPipeline) with around 1000 runs. When accessing it logged in it needs roughly 12 seconds. Not logged in the load is almost instantaneous.

            (Jenkins 2.90, BlueOcean 1.3.2)

            A pipeline job with 55 runs takes around 4 seconds to load using blue ocean. 

            ---------------------

            I got a considerable performance improvement once I deleted a couple of old builds (We had 30 stored for now). After reducing this to 7, the Pipeline Build now loads way faster (around 1-2 seconds)

             

            So I assume some time access through a large number of build (and/or files) are the problem.

             

            Show
            seckler Steffen Seckler added a comment - - edited I do experience the same problem. I have attached the necessary support files. support_2017-11-17_08.03.54.zip   support_2017-11-17_08.04.02.zip It happens for me when accessing a pipeline job (using the non-declarative version, called MardynPipeline) with around 1000 runs. When accessing it logged in it needs roughly 12 seconds. Not logged in the load is almost instantaneous. (Jenkins 2.90, BlueOcean 1.3.2) A pipeline job with 55 runs takes around 4 seconds to load using blue ocean.  --------------------- I got a considerable performance improvement once I deleted a couple of old builds (We had 30 stored for now). After reducing this to 7, the Pipeline Build now loads way faster (around 1-2 seconds)   So I assume some time access through a large number of build (and/or files) are the problem.  
            Hide
            seckler Steffen Seckler added a comment -

            Besides of that I get a rather slow response when I open BlueOcean for the first time in a while. (see attachments) support_2017-11-17_11.26.08.zip

            Show
            seckler Steffen Seckler added a comment - Besides of that I get a rather slow response when I open BlueOcean for the first time in a while. (see attachments)  support_2017-11-17_11.26.08.zip
            Hide
            bksaville Brian Saville added a comment -

            After saying that we hadn't seen it in awhile, I've also been noticing this problem again recently. It takes much more than 10-12 seconds though, more like 2-3 minutes to load initially, and then extremely fast after that. Loading the classic UI is immediate, so this must be something with blue ocean itself.

            I will try to get a support bundle at some point here, but it may take me a bit.

            Show
            bksaville Brian Saville added a comment - After saying that we hadn't seen it in awhile, I've also been noticing this problem again recently. It takes much more than 10-12 seconds though, more like 2-3 minutes to load initially, and then extremely fast after that. Loading the classic UI is immediate, so this must be something with blue ocean itself. I will try to get a support bundle at some point here, but it may take me a bit.
            Hide
            michaelneale Michael Neale added a comment -

            thanks Steffen Seckler Brian Saville - that is most useful. 

            So clearly there is come caching going on, HOWEVER, given classic doesn't have this, it means that blue ocean is touching some data that it shouldn't, and given it seems to be proportional to the run history, it breaks paging of the data somehow. in the past we saw this with actions... 

            Any support bundles help, at least to know if there is some plugin causing this... 

             

            The key is that we want to have test coverage for this if possible: we will want to have a test case that creates 1000+ runs and possibly times the loading of it relative to classic so it isn't 10x slower... but this sounds error prone test wise... so may need to think other strategies to verify this... 

            Show
            michaelneale Michael Neale added a comment - thanks Steffen Seckler Brian Saville - that is most useful.  So clearly there is come caching going on, HOWEVER, given classic doesn't have this, it means that blue ocean is touching some data that it shouldn't, and given it seems to be proportional to the run history, it breaks paging of the data somehow. in the past we saw this with actions...  Any support bundles help, at least to know if there is some plugin causing this...    The key is that we want to have test coverage for this if possible: we will want to have a test case that creates 1000+ runs and possibly times the loading of it relative to classic so it isn't 10x slower... but this sounds error prone test wise... so may need to think other strategies to verify this... 
            Hide
            michaelneale Michael Neale added a comment -

            Steffen Seckler so - do you see it on the dashboard or activity/pipeline page only? 

            Show
            michaelneale Michael Neale added a comment - Steffen Seckler so - do you see it on the dashboard or activity/pipeline page only? 
            Hide
            seckler Steffen Seckler added a comment -

            Hmm, I only see this on the dashboard and the activity side, branches and pull requests work fine. But to be fair, we don't have multi branch or PR enabled, so these pages are normally blank.

            for activity side, this is not limited to normal pipeline builds, but also freestyle jobs.

            works fine:

            • new pipeline etc., but there no file touching should happen anyways. 
            • viewing specific builds of a pipeline

             

            what i could imagine, is that all builds of a pipeline are already cached when loading a pipeline. as we use a network drive, which is not the fastest, this might be the underlying problem...

            Show
            seckler Steffen Seckler added a comment - Hmm, I only see this on the dashboard and the activity side, branches and pull requests work fine. But to be fair, we don't have multi branch or PR enabled, so these pages are normally blank. for activity side, this is not limited to normal pipeline builds, but also freestyle jobs. works fine: new pipeline etc., but there no file touching should happen anyways.  viewing specific builds of a pipeline   what i could imagine, is that all builds of a pipeline are already cached when loading a pipeline. as we use a network drive, which is not the fastest, this might be the underlying problem...
            Hide
            bksaville Brian Saville added a comment -

            I should also note that we limit all of our builds to only keep 10 at a time. So a large number of runs shouldn't be related to us, unless a "large number" is 10.

            Show
            bksaville Brian Saville added a comment - I should also note that we limit all of our builds to only keep 10 at a time. So a large number of runs shouldn't be related to us, unless a "large number" is 10.
            Hide
            michaelneale Michael Neale added a comment -

            hrm ok - so it isn't quite clear which problem we are talking about - other than logged in is sometimes slower... 

            Show
            michaelneale Michael Neale added a comment - hrm ok - so it isn't quite clear which problem we are talking about - other than logged in is sometimes slower... 
            Hide
            bksaville Brian Saville added a comment -

            Honestly, it could be quite a few things contributing to this. Setting up some kind of test for this against classic seems reasonable, although fragile.

            Show
            bksaville Brian Saville added a comment - Honestly, it could be quite a few things contributing to this. Setting up some kind of test for this against classic seems reasonable, although fragile.
            Hide
            bksaville Brian Saville added a comment -

            I attached a support bundle (support_2017-11-20_23.21.08.zip).

            Show
            bksaville Brian Saville added a comment - I attached a support bundle (support_2017-11-20_23.21.08.zip).
            Hide
            kshultz Karl Shultz added a comment -

            Testing Notes:

            • We should look at performance testing of Blue Ocean, especially in environments which have large numbers (hundreds? Over 1000?) of projects and/or branches.
            • These performance tests could be designed to compare the same basic workflows between Classic and Blue Ocean...although that feels like it'll be a needy, fragile test.
            Show
            kshultz Karl Shultz added a comment - Testing Notes: We should look at performance testing of Blue Ocean, especially in environments which have large numbers (hundreds? Over 1000?) of projects and/or branches. These performance tests could be designed to compare the same basic workflows between Classic and Blue Ocean...although that feels like it'll be a needy, fragile test.
            Hide
            michaelneale Michael Neale added a comment -

            Karl Shultz that would be most awesome - for extra points if we could delta versions (not easy though), could be good. 

            Dashboard loading, result page loading and activity screen are known problem areas. 

            Show
            michaelneale Michael Neale added a comment - Karl Shultz that would be most awesome - for extra points if we could delta versions (not easy though), could be good.  Dashboard loading, result page loading and activity screen are known problem areas. 
            Hide
            michaelneale Michael Neale added a comment - - edited

            Brian Saville Steffen Seckler I am having trouble making sense of the support bundles just now ("slow requests" aren't showing up in them"). If you are able to experiment then disabling the blue ocean JIRA plugin (new since 1.3) may make sense (although if you saw this with 1.2 - then disregard). 

            EDIT: also if you could turn on "slow requests" as an option when you generate the support bundle, would be helpful. 

             

             

            Vivek Pandey I am seeing some stack traces like this

             

            "Handling GET /mardyn/blue/rest/organizations/jenkins/pipelines/PIPE_NAME/runs/ from 0:0:0:0:0:0:0:1 : qtp997608398-19" id=19 (0x13) state=RUNNABLE cpu=92%
            at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
            at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:242)
            at java.io.File.exists(File.java:819)
            at hudson.model.AbstractBuild.calcChangeSet(AbstractBuild.java:871)
            at hudson.model.AbstractBuild.getChangeSet(AbstractBuild.java:843)
            at hudson.model.AbstractBuild.getChangeSets(AbstractBuild.java:857)
            at jenkins.scm.RunWithSCM.hasParticipant(RunWithSCM.java:146)

            which implies it is our friend calcChangeSet hurting things again - this is non multi branch so isn't the multibranch activity calculation but just listing a page of runs (ci.jenkins.io DOES use multibranch on the core jenkins project, and is very slow, but may be same thing... if we can get s support core from them too may see that). More investigation needed... 

            Show
            michaelneale Michael Neale added a comment - - edited Brian Saville Steffen Seckler  I am having trouble making sense of the support bundles just now ("slow requests" aren't showing up in them"). If you are able to experiment then disabling the blue ocean JIRA plugin (new since 1.3) may make sense (although if you saw this with 1.2 - then disregard).  EDIT: also if you could turn on "slow requests" as an option when you generate the support bundle, would be helpful.      Vivek Pandey I am seeing some stack traces like this   "Handling GET /mardyn/blue/ rest /organizations/jenkins/pipelines/PIPE_NAME/runs/ from 0:0:0:0:0:0:0:1 : qtp997608398-19" id=19 (0x13) state=RUNNABLE cpu=92% at java.io.UnixFileSystem.getBooleanAttributes0(Native Method) at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:242) at java.io.File.exists(File.java:819) at hudson.model.AbstractBuild.calcChangeSet(AbstractBuild.java:871) at hudson.model.AbstractBuild.getChangeSet(AbstractBuild.java:843) at hudson.model.AbstractBuild.getChangeSets(AbstractBuild.java:857) at jenkins.scm.RunWithSCM.hasParticipant(RunWithSCM.java:146) which implies it is our friend calcChangeSet hurting things again - this is non multi branch so isn't the multibranch activity calculation but just listing a page of runs (ci.jenkins.io DOES use multibranch on the core jenkins project, and is very slow, but may be same thing... if we can get s support core from them too may see that). More investigation needed... 
            Hide
            jamesdumay James Dumay added a comment -

            Let me check that the JIRA plugin isn't actually doing something dumb.

            Show
            jamesdumay James Dumay added a comment - Let me check that the JIRA plugin isn't actually doing something dumb.
            Hide
            michaelneale Michael Neale added a comment -

            So Keith Zantow has a ticket for the user info fetching that may be slowing down the logged in case. 

            Show
            michaelneale Michael Neale added a comment - So Keith Zantow has a ticket for the user info fetching that may be slowing down the logged in case. 
            Hide
            kzantow Keith Zantow added a comment -

            This might just be something in my brain, I can't find a related ticket. However, I am fairly sure that the issue has to do with fetching one specific type of user data.

            Show
            kzantow Keith Zantow added a comment - This might just be something in my brain, I can't find a related ticket. However, I am fairly sure that the issue has to do with fetching one specific type of user data.
            Hide
            bksaville Brian Saville added a comment -

            Michael Neale, our problem definitely only occurs when I'm logged in. I don't believe others see it when they are logged in, possibly only system administrators. I would be getting a lot more complaints if either of those were not true since we have 100s of internal users that are not exactly forgiving

            Show
            bksaville Brian Saville added a comment - Michael Neale , our problem definitely only occurs when I'm logged in. I don't believe others see it when they are logged in, possibly only system administrators. I would be getting a lot more complaints if either of those were not true since we have 100s of internal users that are not exactly forgiving
            Hide
            michaelneale Michael Neale added a comment -

            ok Brian Saville good to know, so the not logged in users are ok (presumably). do you have a lot of favourites by any chance? A lot of runs? is it a specific pipeline that loads slow while others (with less runs) load fast? 

             

            OK I have removed that other ticket, this still seems primarily related to being logged in. Brian Saville do you have a support bundle - sorry can't seem to tell the files apart (if you do, can you link which one is yours in the comment?)

            Show
            michaelneale Michael Neale added a comment - ok Brian Saville good to know, so the not logged in users are ok (presumably). do you have a lot of favourites by any chance? A lot of runs? is it a specific pipeline that loads slow while others (with less runs) load fast?    OK I have removed that other ticket, this still seems primarily related to being logged in. Brian Saville do you have a support bundle - sorry can't seem to tell the files apart (if you do, can you link which one is yours in the comment?)
            Hide
            schulzha Hans Schulz added a comment -

            Michael Neale for me the issue was definitely the favorites. The autofavorite is going crazy on me, which is why I also filed https://issues.jenkins-ci.org/browse/JENKINS-47214.
            When I remove the favorites, the performance is about the same (though it still could be faster ). I have also checked with some users that their performance is the same (they have no favorites).

            Show
            schulzha Hans Schulz added a comment - Michael Neale for me the issue was definitely the favorites. The autofavorite is going crazy on me, which is why I also filed  https://issues.jenkins-ci.org/browse/JENKINS-47214 . When I remove the favorites, the performance is about the same (though it still could be faster ). I have also checked with some users that their performance is the same (they have no favorites).
            Hide
            bksaville Brian Saville added a comment - - edited

            Michael Neale, I don't have any favorites - not even sure how to add them to be honest We only keep 10 runs at a time for almost all of our jobs, so it wouldn't be a large number of runs (unless you count the build # being higher, as in 100-300, but I've never noticed this affecting load time, only whether I am logged in or not).

            I will add a support bundle again - I didn't have slow requests in my last one, but I just can't do it today (on call until tomorrow afternoon).

            Show
            bksaville Brian Saville added a comment - - edited Michael Neale , I don't have any favorites - not even sure how to add them to be honest We only keep 10 runs at a time for almost all of our jobs, so it wouldn't be a large number of runs (unless you count the build # being higher, as in 100-300, but I've never noticed this affecting load time, only whether I am logged in or not). I will add a support bundle again - I didn't have slow requests in my last one, but I just can't do it today (on call until tomorrow afternoon).
            Hide
            jamesdumay James Dumay added a comment -

            Brian Saville there is a new version of the autofavorite plugin that allows you to turn off the auto favorite behaviour in the user profile settings. Look for 1.2.0 in a few hours - should be out soon.

            Show
            jamesdumay James Dumay added a comment - Brian Saville there is a new version of the autofavorite plugin that allows you to turn off the auto favorite behaviour in the user profile settings. Look for 1.2.0 in a few hours - should be out soon.
            Hide
            bksaville Brian Saville added a comment - - edited

            Michael Neale and James Dumay, I really should have followed up with Hans Schulz problem earlier - that was precisely my problem as well. I just never configured anything with favorites and didn't realize there was an autofavorite process of any kind. It's nice I'll be able to turn it off (although it now requires a newer jenkins version..sigh, I'm on 2.73.2, just one point release short), but I'm worried that others will run into this as well and won't know. Is there a way to globally disable the autofavorite process?

            We use another system to see the bigger picture of what's going on with all of our jobs, so favorites aren't useful to us at all.

            Show
            bksaville Brian Saville added a comment - - edited Michael Neale and James Dumay , I really should have followed up with Hans Schulz problem earlier - that was precisely my problem as well. I just never configured anything with favorites and didn't realize there was an autofavorite process of any kind. It's nice I'll be able to turn it off (although it now requires a newer jenkins version..sigh, I'm on 2.73.2, just one point release short), but I'm worried that others will run into this as well and won't know. Is there a way to globally disable the autofavorite process? We use another system to see the bigger picture of what's going on with all of our jobs, so favorites aren't useful to us at all.
            Hide
            michaelneale Michael Neale added a comment -

            ok some more investiagtion Brian Saville Ivan Meredith

            So it looks like the favourite thing has been solved with a release of autofav plugin - allowing people to opt out (or system wide opt out). You still need to remove the favourites, but I think with this the logged in slowness case may be solved..

             

             

            However, this does leave the general slowness around activity screens. Some new facts: 

             

            Karl Shultz do you happen to have a test set of data that could work with 1.1, 1.2, 1.3 to compare activity screen load times for a multibranch project (large number of branches perhaps?), so we know this is a regression or not? 

             

            Show
            michaelneale Michael Neale added a comment - ok some more investiagtion Brian Saville Ivan Meredith :  So it looks like the favourite thing has been solved with a release of autofav plugin - allowing people to opt out (or system wide opt out). You still need to remove the favourites, but I think with this the logged in slowness case may be solved..     However, this does leave the general slowness around activity screens. Some new facts:  ci.jenkins.io has this slowness on the core pipeline, but the number of runs doesn't seem too high, so it may not be related to number of runs (others seem to be reporting similar) ci.jenkins.io has a lot of branches on the core pipeline, so some of the slowness may be to do with that ci.blueocean.io also has a similar number of runs, and many branches, but doesn't seem slow at all. so there may be some plugin/config of jenkins.io that changes its behavior  You can see this on this URL: https://ci.blueocean.io/blue/rest/organizations/jenkins/pipelines/blueocean/runs/?start=0&limit=26 loads pretty fast However: https://ci.jenkins.io/blue/rest/organizations/jenkins/pipelines/Core/jenkins/runs/?start=0&limit=26  is really slow This seems to be a regression on 1.3 (but still not clear) If you reduce the number of rows returned on ci.jenkins.io it is fast:  https://ci.jenkins.io/blue/rest/organizations/jenkins/pipelines/Core/jenkins/runs/?start=0&limit=5 So probably not related to pagination? Maybe just N branches * 26 rows of each to be fetched? Why is it so much faster just for that small diff of data?    Karl Shultz do you happen to have a test set of data that could work with 1.1, 1.2, 1.3 to compare activity screen load times for a multibranch project (large number of branches perhaps?), so we know this is a regression or not?   
            Hide
            michaelneale Michael Neale added a comment -

            Baptiste Mathus is going to get a stacktrace while slowness is happening. 
            James Nord has suggested it may be to do with there being many unstable runs of a specific pipeline, and it has to (erroneosly) recursively load the runs (this may explain why in some cases it is bad, but not others). 

            Both of them confirm it seem to be 1.3 that is generally worse  

            Show
            michaelneale Michael Neale added a comment - Baptiste Mathus is going to get a stacktrace while slowness is happening.  James Nord has suggested it may be to do with there being many unstable runs of a specific pipeline, and it has to (erroneosly) recursively load the runs (this may explain why in some cases it is bad, but not others).  Both of them confirm it seem to be 1.3 that is generally worse  
            Hide
            kshultz Karl Shultz added a comment -

            Michael Neale, apologies, I missed your question from a few days ago. The short answer is no, I don't have a particular dataset, but do have some ideas:

            • We could pick from any number repositories to fork in order to create a test data set. In JENKINS-45372, James Nord was using Apache Maven. That one's got 29 branches in it and would give a test instance plenty of "stuff to do," so to speak. But it's not hundreds of branches.
            • We could create a test repo programatically, and test performance that way. Perhaps as an offshoot of BitbucketServerTest. That might make things nicely self-contained. And would give us some additional Bitbucket coverage implicitly, which might be a welcome "side effect."

            I thought I'd created an Improvement ticket for at least starting the ball rolling on performance testing of BO, but I don't see it. I'll do so later today.

            Show
            kshultz Karl Shultz added a comment - Michael Neale , apologies, I missed your question from a few days ago. The short answer is no, I don't have a particular dataset, but do have some ideas: We could pick from any number repositories to fork in order to create a test data set. In JENKINS-45372 , James Nord was using Apache Maven. That one's got 29 branches in it and would give a test instance plenty of "stuff to do," so to speak. But it's not hundreds of branches. We could create a test repo programatically, and test performance that way. Perhaps as an offshoot of BitbucketServerTest . That might make things nicely self-contained. And would give us some additional Bitbucket coverage implicitly, which might be a welcome "side effect." I thought I'd created an Improvement ticket for at least starting the ball rolling on performance testing of BO, but I don't see it. I'll do so later today.
            Hide
            imeredith Ivan Meredith added a comment -

            After spending time investigating the activity API, I have hit a roadblock.

            The performance issues comes while the data is being serialised and sent to the client. This is as opposed to before serialisation happens while we are generating the iterators of BlueRuns. In the past when we have had performance issues it has been during the pre-serialization step that has been problematic.

            This problem only seems to affect multibranch activity api. And only for some multibranch pipelines. On the blueocean ci server, we have 400ish branches and many builds with no issue, however the jenkinsci ci server has issues with <150 branches and probably less builds.

            Because multibranch activity is loading the 25 latest runs from all branches, it may be loading the last run from 25 different branches. Keith Zantow pointed out that it might be a slowness related to loading runs from so many branches, which would not be as obvious if for example the last 25 runs only came from 2 branches despite many more branches existing. This could explain the difference between what I see on the CI servers.

            I'm not sure what is the best way forward here. I see [at least] 3 alternatives

            • Something can be fixed in core or pipeline to make this faster assuming the issue is there.
            • Add some caching to the activity api. Maybe use an H2 database or just basic in memory cache.
            • Make the branches page the default page instead of activity for multibranch pipelines. This doesn't fix anything but maybe makes it less annoying in the default case.
            Show
            imeredith Ivan Meredith added a comment - After spending time investigating the activity API, I have hit a roadblock. The performance issues comes while the data is being serialised and sent to the client. This is as opposed to before serialisation happens while we are generating the iterators of BlueRuns. In the past when we have had performance issues it has been during the pre-serialization step that has been problematic. This problem only seems to affect multibranch activity api. And only for some multibranch pipelines. On the blueocean ci server, we have 400ish branches and many builds with no issue, however the jenkinsci ci server has issues with <150 branches and probably less builds. Because multibranch activity is loading the 25 latest runs from all branches, it may be loading the last run from 25 different branches. Keith Zantow pointed out that it might be a slowness related to loading runs from so many branches, which would not be as obvious if for example the last 25 runs only came from 2 branches despite many more branches existing. This could explain the difference between what I see on the CI servers. I'm not sure what is the best way forward here. I see [at least]  3 alternatives Something can be fixed in core or pipeline to make this faster assuming the issue is there. Add some caching to the activity api. Maybe use an H2 database or just basic in memory cache. Make the branches page the default page instead of activity for multibranch pipelines. This doesn't fix anything but maybe makes it less annoying in the default case.
            Hide
            michaelneale Michael Neale added a comment -

            Vivek Pandey in light of Ivan's comments above - do you have any more ideas? I still don't have concrete info that it is a regression as of 1.3, but it seems a bit slower. Perhaps it is time to bite the bullet and cache activity screen? (the logged in case should be solved by now BTW). 

            Show
            michaelneale Michael Neale added a comment - Vivek Pandey in light of Ivan's comments above - do you have any more ideas? I still don't have concrete info that it is a regression as of 1.3, but it seems a bit slower. Perhaps it is time to bite the bullet and cache activity screen? (the logged in case should be solved by now BTW). 
            Hide
            vivek Vivek Pandey added a comment - - edited

            Bug identified and fixed. PR opened https://github.com/jenkinsci/blueocean-plugin/pull/1632.

             

            Details on what was causing and fix:

            Analyzing har file showed, FavoriteStatePreloader was returning large number of favorite jobs resulting in most time taken responding to loading dashboard. Bug was in `FavoriteContainer.iterator()` not paginating. Fix adds pagination by default.

            Show
            vivek Vivek Pandey added a comment - - edited Bug identified and fixed. PR opened https://github.com/jenkinsci/blueocean-plugin/pull/1632.   Details on what was causing and fix: Analyzing har file showed, FavoriteStatePreloader was returning large number of favorite jobs resulting in most time taken responding to loading dashboard. Bug was in `FavoriteContainer.iterator()` not paginating. Fix adds pagination by default.
            Hide
            kzantow Keith Zantow added a comment -

            I don't see a problem with that PR, but doesn't the page display all favorites? So I'm not sure how paging would fix that.

            Show
            kzantow Keith Zantow added a comment - I don't see a problem with that PR, but doesn't the page display all favorites? So I'm not sure how paging would fix that.
            Hide
            vivek Vivek Pandey added a comment -

            Keith Zantow Currently yes, it fetches all possible favorites. container iterators are supposed to return default page size. If you call favorite API you get default page size so it makes it consistent.

            Favorites are expensive and displaying 100s of favorite is rather pain to user as bunch of them are auto-favorited. I think a new ticket should be opened to add UI pagination support ('Show more') for favorite as well. Current page size is 100, maybe it should be 26 like other things shown in UI?

             

             

            Show
            vivek Vivek Pandey added a comment - Keith Zantow  Currently yes, it fetches all possible favorites. container iterators are supposed to return default page size. If you call favorite API you get default page size so it makes it consistent. Favorites are expensive and displaying 100s of favorite is rather pain to user as bunch of them are auto-favorited. I think a new ticket should be opened to add UI pagination support ('Show more') for favorite as well. Current page size is 100, maybe it should be 26 like other things shown in UI?    
            Hide
            vivek Vivek Pandey added a comment -

            Keith Zantow over to you as discussed. Listing the issues we discussed to fix as part of this improvement:

            • Paginate favorite list on dashboard (my fix fixes that, that is default list of favorite is default page size of 100, possibly it should be 26 like other objects)
            • Evaluate if top level pipeline object can include 'favorite' as boolean value without much impact
            • Minimize 'item' object properties to include only what favorite UI needs (name of pipeline, favorited or not, commitId...)
            • Fix bug in frontend where it calls favorites API even though it already got pre-loaded list of favorites
            Show
            vivek Vivek Pandey added a comment - Keith Zantow over to you as discussed. Listing the issues we discussed to fix as part of this improvement: Paginate favorite list on dashboard (my fix fixes that, that is default list of favorite is default page size of 100, possibly it should be 26 like other objects) Evaluate if top level pipeline object can include 'favorite' as boolean value without much impact Minimize 'item' object properties to include only what favorite UI needs (name of pipeline, favorited or not, commitId...) Fix bug in frontend where it calls favorites API even though it already got pre-loaded list of favorites
            Hide
            michaelneale Michael Neale added a comment -

            Josh McDonald as mentioned - worth taking a look at (I will take a look at test failures too)

            Show
            michaelneale Michael Neale added a comment - Josh McDonald as mentioned - worth taking a look at (I will take a look at test failures too)

              People

              • Assignee:
                nicu Nicolae Pascu
                Reporter:
                schulzha Hans Schulz
              • Votes:
                7 Vote for this issue
                Watchers:
                21 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: