Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56773

High master CPU usage introduced by Blue Ocean 1.14.0

    Details

    • Type: Bug
    • Status: Reopened (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: blueocean-plugin
    • Labels:
      None
    • Environment:
      BlueOcean 1.14.0
      Jenkins 1.168
    • Similar Issues:
    • Released As:
      1.15.0

      Description

      We went from BO 1.13.2 to BO 1.14.0.
      After the upgrade, we have seen a major CPU increase on our Jenkins master, leading to really slow UI and job execution, with threads being locked - see screenshot.

      Our master is configured to not run any job ("only build jobs with label expression matching this node").
      Using Jenkins monitor, we saw that ~90% of the CPU time was consumed by calls to the rest endpoint /blue/rest/organizations/jenkins/computers - see screenshot.

      Looking at the release notes from 1.14.0, it seems that JENKINS-52825 has introduced that endpoint and should be an area to investigate.

      We had to rollback BO to 1.13.2 which does not exhibit that behavior.

        Attachments

          Issue Links

            Activity

            ojacques Olivier Jacques created issue -
            Hide
            ojacques Olivier Jacques added a comment -

            /cc Craig Rodrigues - in case this can indeed be caused by JENKINS-52825.

            Show
            ojacques Olivier Jacques added a comment - /cc Craig Rodrigues - in case this can indeed be caused by JENKINS-52825 .
            halkeye Gavin Mogan made changes -
            Field Original Value New Value
            Link This issue is related to JENKINS-56833 [ JENKINS-56833 ]
            Hide
            halkeye Gavin Mogan added a comment -

            So had a bit of a chat with keith (original executor plugin author) and it was never really designed for more than a proof of concept on how to write a BlueOcean plugin.

            We are pretty sure its caused by https://github.com/jenkinsci/blueocean-plugin/blob/e2dd04dcb3015063a2e31c8d37c90ee5c1ef4d98/blueocean-executor-info/src/main/js/ExecutorInfoService.js#L14 (which will grab executor info every time any pipeline does work)

            So Plans:

            • I think the plan is going to be to drop making the executor plugin as a requirement, it'll have to be manually installed again (Sorry Craig)
            • Put a throttle on the lookup so it doesn't call nearly as often.

            Since there is a work around in place until the next release, and I personally have a few pressing issues to deal with first, i'm going to leave it unassigned, but we'll get it fixed before we do the next release.

            Show
            halkeye Gavin Mogan added a comment - So had a bit of a chat with keith (original executor plugin author) and it was never really designed for more than a proof of concept on how to write a BlueOcean plugin. We are pretty sure its caused by  https://github.com/jenkinsci/blueocean-plugin/blob/e2dd04dcb3015063a2e31c8d37c90ee5c1ef4d98/blueocean-executor-info/src/main/js/ExecutorInfoService.js#L14  (which will grab executor info every time any pipeline does work) So Plans: I think the plan is going to be to drop making the executor plugin as a requirement, it'll have to be manually installed again (Sorry Craig) Put a throttle on the lookup so it doesn't call nearly as often. Since there is a work around in place until the next release, and I personally have a few pressing issues to deal with first, i'm going to leave it unassigned, but we'll get it fixed before we do the next release.
            Hide
            ojacques Olivier Jacques added a comment -

            Thanks Gavin Mogan.

            Show
            ojacques Olivier Jacques added a comment - Thanks Gavin Mogan .
            Show
            halkeye Gavin Mogan added a comment - https://github.com/jenkinsci/blueocean-plugin/pull/1947
            Hide
            dgoetsch Devyn Goetsch added a comment -

            This impacted our company and we had to do the same rollback.

            we went from 1.6.x to 1.14.0, the next work day jenkins became unresponsive under load, and our jvm monitoring told us that the blue ocean rest computers endpoint was using most of the cpu time.

            We rolled back to 1.13.2 and disabled the Blue Ocean Executor Info plugin and our Jenkins master is far more responsive and page loads for blue ocean are much lower.

            Our builds have 3 stages, the first one is a single stage, the second one has 3 parallel stages, and the third one has 50 parallel stages, 32 of which are computed dynamically and launched with the "node" step.

            Please revert the change to make the Blue Ocean Executor Info plugin a required dependency.  I'm sure it would be okay on a smaller system, but for a system like ours it is untenable.

            Show
            dgoetsch Devyn Goetsch added a comment - This impacted our company and we had to do the same rollback. we went from 1.6.x to 1.14.0, the next work day jenkins became unresponsive under load, and our jvm monitoring told us that the blue ocean rest computers endpoint was using most of the cpu time. We rolled back to 1.13.2 and disabled the Blue Ocean Executor Info plugin and our Jenkins master is far more responsive and page loads for blue ocean are much lower. Our builds have 3 stages, the first one is a single stage, the second one has 3 parallel stages, and the third one has 50 parallel stages, 32 of which are computed dynamically and launched with the "node" step. Please revert the change to make the Blue Ocean Executor Info plugin a required dependency.  I'm sure it would be okay on a smaller system, but for a system like ours it is untenable.
            halkeye Gavin Mogan made changes -
            Link This issue is related to JENKINS-56833 [ JENKINS-56833 ]
            halkeye Gavin Mogan made changes -
            Status Open [ 1 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            Released As 1.15.0
            halkeye Gavin Mogan made changes -
            Assignee Gavin Mogan [ halkeye ]
            sorenfriis Søren Friis made changes -
            Attachment computers_GET.png [ 47921 ]
            Hide
            sorenfriis Søren Friis added a comment -

            We are still seeing a very high amount of CPU time spend servicing the following endpoint on Blue Ocean 1.17.0

            /blue/rest/organizations/jenkins/computers/
            

            computers_GET.png

            I tried blocking the endpoint in my browser, and everything seemed to function normally on Blue Ocean pages.
            Can the polling of this endpoint be disabled or reduced?

            Show
            sorenfriis Søren Friis added a comment - We are still seeing a very high amount of CPU time spend servicing the following endpoint on Blue Ocean 1.17.0 /blue/rest/organizations/jenkins/computers/ computers_GET.png I tried blocking the endpoint in my browser, and everything seemed to function normally on Blue Ocean pages. Can the polling of this endpoint be disabled or reduced?
            Hide
            halkeye Gavin Mogan added a comment -

            disable the executor plugin, its no longer required

            Show
            halkeye Gavin Mogan added a comment - disable the executor plugin, its no longer required
            Hide
            danielbeck Daniel Beck added a comment -

            Removing a plugin from the aggregator/meta plugin doesn't remove the broken functionality from any instances that have it installed.

            Like ci.jenkins.io which got shit for the last several weeks for its recent stability problems. Imagine my surprise seeing dozens of threads hanging in the endpoint mentioned in this issue, and how this issue has been resolved.

            At the very least, admins that generally take care of their instance need to be informed about this with an admin monitor.

            Show
            danielbeck Daniel Beck added a comment - Removing a plugin from the aggregator/meta plugin doesn't remove the broken functionality from any instances that have it installed. Like ci.jenkins.io which got shit for the last several weeks for its recent stability problems. Imagine my surprise seeing dozens of threads hanging in the endpoint mentioned in this issue, and how this issue has been resolved. At the very least, admins that generally take care of their instance need to be informed about this with an admin monitor.
            danielbeck Daniel Beck made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-59706 [ JENKINS-59706 ]
            Hide
            markewaite Mark Waite added a comment -

            As far as I can tell, https://github.com/jenkinsci/blueocean-plugin/releases/tag/blueocean-parent-1.21.0 includes this change. Thanks Jesse Glick for that pull request! I think this can be marked 'Closed' in release blue ocean 1.21.0 with a code fix to prevent the issue.

            Can one of the users that encountered this problem confirm that it is now resolved?

            Show
            markewaite Mark Waite added a comment - As far as I can tell, https://github.com/jenkinsci/blueocean-plugin/releases/tag/blueocean-parent-1.21.0 includes this change. Thanks Jesse Glick for that pull request! I think this can be marked 'Closed' in release blue ocean 1.21.0 with a code fix to prevent the issue. Can one of the users that encountered this problem confirm that it is now resolved?
            Hide
            mwos Michał Woś added a comment - - edited

            We use Blue Ocean and Blue Ocean Executor Info plugins in version 1.22.0 and still have a problem with 

            ```

            /blue/rest/organizations/jenkins/computers/

            ```

            endpoint. It is called very often and hangs for ~5sec each time. Not sure why informations about all executors is needed in the first place when opened blue ocean for a given job and a run. Those informations do not seem to be utilised anywhere in the view.

            Show
            mwos Michał Woś added a comment - - edited We use Blue Ocean and Blue Ocean Executor Info plugins in version 1.22.0 and still have a problem with  ``` /blue/rest/organizations/jenkins/computers/ ``` endpoint. It is called very often and hangs for ~5sec each time. Not sure why informations about all executors is needed in the first place when opened blue ocean for a given job and a run. Those informations do not seem to be utilised anywhere in the view.
            Hide
            markewaite Mark Waite added a comment -

            Michał Woś sorry that you've encountered the problem. Per the comment from Gavin Mogan, the e Blue Ocean Executor Info plugin is no longer required. You might be able to resolve the issue on your installation by removing the Blue Ocean Executor Info plugin. It is not installed on ci.jenkins.io (likely because Daniel Beck removed it). I was able to remove it from my Jenkins instance without seeing any harmful side effects.

            I agree with Daniel that it would be much better to have an administrative monitor or a Jenkins Health Advisor warning that Blue Ocean Executor Info is not required and may cause performance issues.

            Show
            markewaite Mark Waite added a comment - Michał Woś sorry that you've encountered the problem. Per the comment from Gavin Mogan , the e Blue Ocean Executor Info plugin is no longer required. You might be able to resolve the issue on your installation by removing the Blue Ocean Executor Info plugin. It is not installed on ci.jenkins.io (likely because Daniel Beck removed it). I was able to remove it from my Jenkins instance without seeing any harmful side effects. I agree with Daniel that it would be much better to have an administrative monitor or a Jenkins Health Advisor warning that Blue Ocean Executor Info is not required and may cause performance issues.
            halkeye Gavin Mogan made changes -
            Assignee Gavin Mogan [ halkeye ]
            Hide
            baychris Chris Bayliss added a comment -

            Same issue. Enabling the executor plugin made our Jenkins instance unusable for a couple of days until we tracked the issue down to this bug and disabled it. Our Jenkins master took a massive performance hit with this plugin enabled. Please remove it if it's not required. 

            Show
            baychris Chris Bayliss added a comment - Same issue. Enabling the executor plugin made our Jenkins instance unusable for a couple of days until we tracked the issue down to this bug and disabled it. Our Jenkins master took a massive performance hit with this plugin enabled. Please remove it if it's not required. 
            Hide
            halkeye Gavin Mogan added a comment -

            remove it from what?

            Show
            halkeye Gavin Mogan added a comment - remove it from what?
            Hide
            damien_coraboeuf Damien Coraboeuf added a comment -

            Remove the plugin from your Jenkins installation.

            Show
            damien_coraboeuf Damien Coraboeuf added a comment - Remove the plugin from your Jenkins installation.
            Hide
            danielbeck Daniel Beck added a comment - - edited

            It's no longer a dependency in recent BO releases. You can just click the buttons.

            I don't think the Jenkins project should be able to decide what gets uninstalled on people's instances.

            Show
            danielbeck Daniel Beck added a comment - - edited It's no longer a dependency in recent BO releases. You can just click the buttons. I don't think the Jenkins project should be able to decide what gets uninstalled on people's instances.

              People

              • Assignee:
                Unassigned
                Reporter:
                ojacques Olivier Jacques
              • Votes:
                6 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated: