Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-18377

Improve the performance when listing many jobs (GSoC 2019, coding phase 2)

    XMLWordPrintable

    Details

    • Epic Name:
      RoleStrategy-Performance
    • Similar Issues:

      Description

      This affects a jenkins installation with 750+ jobs.

      When loading the "Overview" to list all jobs, the loading time is over 60 seconds.

      With the role-strategy plugin disabled, the loading time goes down to 5 seconds.

        Attachments

          Issue Links

            Activity

            Hide
            croesus Croesus Kall added a comment -

            Profiling shows that the following uses a lot of CPU time

            com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$2perform()

            And this also seems to relate to:
            https://issues.jenkins-ci.org/browse/JENKINS-17122

            Show
            croesus Croesus Kall added a comment - Profiling shows that the following uses a lot of CPU time com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$2perform() And this also seems to relate to: https://issues.jenkins-ci.org/browse/JENKINS-17122
            Hide
            walterk82 Walter Kacynski added a comment -

            I am also experiencing this problem, but not with that many jobs. This started occurring after an upgrade from Jenkins 1.510 to 1.521

            Show
            walterk82 Walter Kacynski added a comment - I am also experiencing this problem, but not with that many jobs. This started occurring after an upgrade from Jenkins 1.510 to 1.521
            Hide
            walterk82 Walter Kacynski added a comment -

            I am increasing the priority of this issue since this prevents upgrades to newer releases of Jenkins.

            Show
            walterk82 Walter Kacynski added a comment - I am increasing the priority of this issue since this prevents upgrades to newer releases of Jenkins.
            Hide
            walterk82 Walter Kacynski added a comment -

            I linked JENKINS-17122 to see if it has any bearing on this problem.

            Show
            walterk82 Walter Kacynski added a comment - I linked JENKINS-17122 to see if it has any bearing on this problem.
            Hide
            ctapobep Stanislav Bashkyrtsev added a comment - - edited

            I've raised issues with detailed analysis: JENKINS-18721 JENKINS-18723 They might be the reason of current problem. RBAC is what makes the problem visible, but the problem itself is not in RBAC (at least that was what my analysis showed to me), but in ListView.

            Show
            ctapobep Stanislav Bashkyrtsev added a comment - - edited I've raised issues with detailed analysis: JENKINS-18721 JENKINS-18723 They might be the reason of current problem. RBAC is what makes the problem visible, but the problem itself is not in RBAC (at least that was what my analysis showed to me), but in ListView.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Seems that I'll experience this issue soon

            Regarding views...
            It is possible to implement caching of the permissions, which will reduce pages loading time. I'll try to implement such cache in the next version of the plugin. Anyway, please feel free to contribute.

            Show
            oleg_nenashev Oleg Nenashev added a comment - Seems that I'll experience this issue soon Regarding views... It is possible to implement caching of the permissions, which will reduce pages loading time. I'll try to implement such cache in the next version of the plugin. Anyway, please feel free to contribute.
            Hide
            gsash Georg Sash added a comment -

            I used to have the same problem. But I did not encounter the problem in recent Jenkins versions. so I guess this issue is obsolete.

            Show
            gsash Georg Sash added a comment - I used to have the same problem. But I did not encounter the problem in recent Jenkins versions. so I guess this issue is obsolete.
            Hide
            croesus Croesus Kall added a comment -

            I still have this problem - so I wouldn't say it was obsolete.

            Show
            croesus Croesus Kall added a comment - I still have this problem - so I wouldn't say it was obsolete.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Fixes in the core (JENKINS-18721) are available since 1.532.
            It partially resolves the issue, but I confirm that there are many improvements to be done inside the plugin.

            I'm going to keep the issue.
            BTW, the priority could be decreased if there is no critical performance complains.

            As a workaround, you can use RoleMacros (e.g. from the Ownership plugin). They greatly reduce the calculation speed due to the internal caching

            Show
            oleg_nenashev Oleg Nenashev added a comment - Fixes in the core ( JENKINS-18721 ) are available since 1.532. It partially resolves the issue, but I confirm that there are many improvements to be done inside the plugin. I'm going to keep the issue. BTW, the priority could be decreased if there is no critical performance complains. As a workaround, you can use RoleMacros (e.g. from the Ownership plugin). They greatly reduce the calculation speed due to the internal caching
            Hide
            croesus Croesus Kall added a comment -

            Definitely faster loading now. Loading time has gone down from over 60 seconds to around 15 seconds.

            But for my setup it takes 5 seconds without role-strategy, so there is still room for improvement.

            Show
            croesus Croesus Kall added a comment - Definitely faster loading now. Loading time has gone down from over 60 seconds to around 15 seconds. But for my setup it takes 5 seconds without role-strategy, so there is still room for improvement.
            Hide
            raphael_unique Raphaël UNIQUE added a comment - - edited

            We have the same problem
            non administrators user experience this issue, each action take a long time.
            for administrators, all is realy faster

            Show
            raphael_unique Raphaël UNIQUE added a comment - - edited We have the same problem non administrators user experience this issue, each action take a long time. for administrators, all is realy faster
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            @Raphael
            It does not make sense to add new comments w/o providing RoleStrategy configurations (corner-cases, etc.) for slow installations.

            Show
            oleg_nenashev Oleg Nenashev added a comment - @Raphael It does not make sense to add new comments w/o providing RoleStrategy configurations (corner-cases, etc.) for slow installations.
            Hide
            raphael_unique Raphaël UNIQUE added a comment - - edited

            some details about the installation:

            • 1300 jobs
            • 27 views
            • AD contain 14 000 users
            • 4 global roles assigned to
              • 21 AD groups (max 100 users)
              • 14 users
            • 38 project roles asigned to
              • 19 AD groups (max 100 users)
              • 11 users

            for non administrators users,each navigation between views can take up to 70 seconds
            for administrators, i takes only 5 second

            Show
            raphael_unique Raphaël UNIQUE added a comment - - edited some details about the installation: 1300 jobs 27 views AD contain 14 000 users 4 global roles assigned to 21 AD groups (max 100 users) 14 users 38 project roles asigned to 19 AD groups (max 100 users) 11 users for non administrators users,each navigation between views can take up to 70 seconds for administrators, i takes only 5 second
            Hide
            arost Alexander Ost added a comment -

            This issue also heavily affects the search box – with role-based auth active (and approx 2500 jobs defined), the search box is barely usable (slow update, long time to feedback). After switching back to matrix-based authentication, the search box works fine.

            As previously mentioned, Job-listing is affected as well, but also access to other pages is much slower ("All" view loading takes about twice as long, master node display takes four times longer when RBAC is active).

            Running LTS Jenkins 1.609.2.

            Show
            arost Alexander Ost added a comment - This issue also heavily affects the search box – with role-based auth active (and approx 2500 jobs defined), the search box is barely usable (slow update, long time to feedback). After switching back to matrix-based authentication, the search box works fine. As previously mentioned, Job-listing is affected as well, but also access to other pages is much slower ("All" view loading takes about twice as long, master node display takes four times longer when RBAC is active). Running LTS Jenkins 1.609.2.
            Hide
            croesus Croesus Kall added a comment -

            With the new release, 2.3.0, the performance is worse than ever (Using jenkins 2.8)
            After downgrading to 2.2.0, things were ok.

            Show
            croesus Croesus Kall added a comment - With the new release, 2.3.0, the performance is worse than ever (Using jenkins 2.8) After downgrading to 2.2.0, things were ok.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Croesus Kall Caching of roles should have helped, but maybe there're some downsides.
            Do you have any metrics from your setup?

            Show
            oleg_nenashev Oleg Nenashev added a comment - Croesus Kall Caching of roles should have helped, but maybe there're some downsides. Do you have any metrics from your setup?
            Hide
            croesus Croesus Kall added a comment -

            Oleg Nenashev Jenkins-wise I have this setup:
            Jobs: about 1900
            Global Roles: 4
            Project roles: 12
            Users: 25

            When I updated to 2.3.0, jenkins became quite slow and the load statistics for the machine were very high.

            Show
            croesus Croesus Kall added a comment - Oleg Nenashev Jenkins-wise I have this setup: Jobs: about 1900 Global Roles: 4 Project roles: 12 Users: 25 When I updated to 2.3.0, jenkins became quite slow and the load statistics for the machine were very high.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Croesus Kall See 2.3.2 changelog

            Show
            oleg_nenashev Oleg Nenashev added a comment - Croesus Kall See 2.3.2 changelog
            Hide
            albernhagen Andy Bernhagen added a comment -

            I also see slow performance in large environments, even on the 2.3.2 upgrade.

            Show
            albernhagen Andy Bernhagen added a comment - I also see slow performance in large environments, even on the 2.3.2 upgrade.
            Hide
            wouterh Wouter Hünd added a comment - - edited

            Oleg Nenashev I'm also still seeing poor performance with a large amount of jobs mapping against ldap roles, even with 2.3.2. Enabling the plugin increases response times of listings by 5-10 seconds.

            Show
            wouterh Wouter Hünd added a comment - - edited Oleg Nenashev I'm also still seeing poor performance with a large amount of jobs mapping against ldap roles, even with 2.3.2. Enabling the plugin increases response times of listings by 5-10 seconds.
            Hide
            olivergondza Oliver Gondža added a comment -
            Show
            olivergondza Oliver Gondža added a comment - Related PR submitted https://github.com/jenkinsci/role-strategy-plugin/pull/28/
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oliver Gondža
            Path:
            src/main/java/com/michelin/cio/hudson/plugins/rolestrategy/Role.java
            http://jenkins-ci.org/commit/role-strategy-plugin/161c0358179a4c845d688b3e42db09f8aa91739d
            Log:
            JENKINS-18377 Cache Role#hashCode to speed up RoleMap#getRolesHavingPermission

            I have experienced severe slowdown caused by following stacktrace:
            at java.util.AbstractSet.hashCode(AbstractSet.java:126)
            at com.michelin.cio.hudson.plugins.rolestrategy.Role.hashCode(Role.java:149)
            at java.util.HashMap.hash(HashMap.java:338)
            at java.util.HashMap.put(HashMap.java:611)
            at java.util.HashSet.add(HashSet.java:219)
            at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$1.perform(RoleMap.java:310)
            at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$RoleWalker.walk(RoleMap.java:387)
            at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$RoleWalker.<init>(RoleMap.java:376)
            at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$1.<init>(RoleMap.java:307)
            at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap.getRolesHavingPermission(RoleMap.java:307)
            at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap.hasPermission(RoleMap.java:107)
            at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap.access$000(RoleMap.java:75)
            at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$AclImpl.hasPermission(RoleMap.java:362)

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oliver Gondža Path: src/main/java/com/michelin/cio/hudson/plugins/rolestrategy/Role.java http://jenkins-ci.org/commit/role-strategy-plugin/161c0358179a4c845d688b3e42db09f8aa91739d Log: JENKINS-18377 Cache Role#hashCode to speed up RoleMap#getRolesHavingPermission I have experienced severe slowdown caused by following stacktrace: at java.util.AbstractSet.hashCode(AbstractSet.java:126) at com.michelin.cio.hudson.plugins.rolestrategy.Role.hashCode(Role.java:149) at java.util.HashMap.hash(HashMap.java:338) at java.util.HashMap.put(HashMap.java:611) at java.util.HashSet.add(HashSet.java:219) at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$1.perform(RoleMap.java:310) at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$RoleWalker.walk(RoleMap.java:387) at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$RoleWalker.<init>(RoleMap.java:376) at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$1.<init>(RoleMap.java:307) at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap.getRolesHavingPermission(RoleMap.java:307) at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap.hasPermission(RoleMap.java:107) at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap.access$000(RoleMap.java:75) at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$AclImpl.hasPermission(RoleMap.java:362)
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            src/main/java/com/michelin/cio/hudson/plugins/rolestrategy/Role.java
            http://jenkins-ci.org/commit/role-strategy-plugin/165538fc1e85d9ff3bae99ab85035f34a4039440
            Log:
            Merge pull request #28 from olivergondza/optimize-RoleMap-getRolesHavingPermission

            JENKINS-18377 Cache Role#hashCode to speed up RoleMap#getRolesHavingPermission

            Compare: https://github.com/jenkinsci/role-strategy-plugin/compare/8dbec9ea93d1...165538fc1e85

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/com/michelin/cio/hudson/plugins/rolestrategy/Role.java http://jenkins-ci.org/commit/role-strategy-plugin/165538fc1e85d9ff3bae99ab85035f34a4039440 Log: Merge pull request #28 from olivergondza/optimize-RoleMap-getRolesHavingPermission JENKINS-18377 Cache Role#hashCode to speed up RoleMap#getRolesHavingPermission Compare: https://github.com/jenkinsci/role-strategy-plugin/compare/8dbec9ea93d1...165538fc1e85
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Unassigning the issue for now. We have added two Role Strategy plugin project ideas to GSoC 2019: https://jenkins.io/projects/gsoc/2019/project-ideas/. If somebody is interested in co-mentoring the ideas (including these tickets), please let us know

            Show
            oleg_nenashev Oleg Nenashev added a comment - Unassigning the issue for now. We have added two Role Strategy plugin project ideas to GSoC 2019: https://jenkins.io/projects/gsoc/2019/project-ideas/ . If somebody is interested in co-mentoring the ideas (including these tickets), please let us know
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            I have converted this issue to EPIC, because there are multiple low-hanging fruits left to be done

            Show
            oleg_nenashev Oleg Nenashev added a comment - I have converted this issue to EPIC, because there are multiple low-hanging fruits left to be done
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Abhyudaya Sharma is currently working on this EPIC as a part of hist GSoC project. https://jenkins.io/projects/gsoc/2019/role-strategy-performance/ .Some stories have been already addressed in the master branch.

            Abhyudaya Sharma could you please take a look at the EPIc and update tasks or add missing ones?

            Show
            oleg_nenashev Oleg Nenashev added a comment - Abhyudaya Sharma is currently working on this EPIC as a part of hist GSoC project.  https://jenkins.io/projects/gsoc/2019/role-strategy-performance/  .Some stories have been already addressed in the master branch. Abhyudaya Sharma could you please take a look at the EPIc and update tasks or add missing ones?
            Hide
            runzexia runze xia added a comment -

            I thought about our authentication model. First we need to get the acl (traversing all the roles), and in the second step to determine whether the role in the acl has the appropriate permissions.
            Can we simplify the acquisition of acl content, we construct the roleMap in the second step, so we can reduce the traversal of some roles.

            This idea may require redesigning the ACL object. The way the ACL is cached may also change.

            Show
            runzexia runze xia added a comment - I thought about our authentication model. First we need to get the acl (traversing all the roles), and in the second step to determine whether the role in the acl has the appropriate permissions. Can we simplify the acquisition of acl content, we construct the roleMap in the second step, so we can reduce the traversal of some roles. This idea may require redesigning the ACL object. The way the ACL is cached may also change.
            Hide
            abhyudaya Abhyudaya Sharma added a comment -

            runze xia I have just committed https://github.com/jenkinsci/role-strategy-plugin/pull/89/commits/a02ea97b0bd512da1973f502c6ab3faca69845d8 for the folder based authorization. This changes how global roles work. Is it something like what you're looking for?

            Show
            abhyudaya Abhyudaya Sharma added a comment - runze xia  I have just committed  https://github.com/jenkinsci/role-strategy-plugin/pull/89/commits/a02ea97b0bd512da1973f502c6ab3faca69845d8  for the folder based authorization. This changes how global roles work. Is it something like what you're looking for?
            Hide
            runzexia runze xia added a comment -

            Abhyudaya Sharma   Yes, it is very similar to the approach I mentioned. If we operate in havePermission func, can we reduce some memory consumption? And we might be very simple to break the loop?

            Show
            runzexia runze xia added a comment - Abhyudaya Sharma    Yes, it is very similar to the approach I mentioned. If we operate in havePermission func, can we reduce some memory consumption? And we might be very simple to break the loop?
            Hide
            jglick Jesse Glick added a comment -

            When loading the "Overview" to list all jobs, the loading time is over 60 seconds.

            Certainly it is good to optimize particular bottlenecks as we find them, but in tandem I would suggest changing the rendering of the dashboard to be progressive so that server-side bottlenecks do not completely prevent display nor block the user from initiating unrelated actions (like clicking on the sidebar). JENKINS-25075 suggests doing this for individual *ListView/column.jelly cells, though you could just as easily render whole rows lazily, ameliorating a bunch of performance problems at once.

            Show
            jglick Jesse Glick added a comment - When loading the "Overview" to list all jobs, the loading time is over 60 seconds. Certainly it is good to optimize particular bottlenecks as we find them, but in tandem I would suggest changing the rendering of the dashboard to be progressive so that server-side bottlenecks do not completely prevent display nor block the user from initiating unrelated actions (like clicking on the sidebar). JENKINS-25075 suggests doing this for individual *ListView/column.jelly cells, though you could just as easily render whole rows lazily, ameliorating a bunch of performance problems at once.

              People

              • Assignee:
                abhyudaya Abhyudaya Sharma
                Reporter:
                croesus Croesus Kall
              • Votes:
                14 Vote for this issue
                Watchers:
                26 Start watching this issue

                Dates

                • Created:
                  Updated: