Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-19623

Jobs throttling causes performance issues in case of enabled security

    Details

    • Similar Issues:

      Description

      • QueueTaskDispatcher locks the queue
      • ThrottleQueueTaskDispatcher invokes Jenkins.Instance().getAllItems() method, which has a security check for the each item
      • Every dispatching attempt causes security check for all jobs in the system
      • Due to big delays, so that dispatching almost blocks the queue and web interfaces

      Workaround - use Hudson.getInstance().getItemMap(), which has not security checks

      "Jenkins cron thread" prio=10 tid=0x0000000005d28000 nid=0x667b runnable [0x0000000041df4000]
      java.lang.Thread.State: RUNNABLE
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$2.perform(RoleMap.java:270)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$RoleWalker.walk(RoleMap.java:327)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$RoleWalker.<init>(RoleMap.java:316)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$2.<init>(RoleMap.java:268)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap.getMatchingRoles(RoleMap.java:268)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap.newMatchingRoleMap(RoleMap.java:224)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy.getACL(RoleBasedAuthorizationStrategy.java:111)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy.getACL(RoleBasedAuthorizationStrategy.java:128)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy.getACL(RoleBasedAuthorizationStrategy.java:123)
      at hudson.model.Job.getACL(Job.java:1324)
      at hudson.model.AbstractItem.hasPermission(AbstractItem.java:448)
      at jenkins.model.Jenkins.getAllItems(Jenkins.java:1397)
      at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.getCategoryProjects(ThrottleQueueTaskDispatcher.java:211)
      at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canRun(ThrottleQueueTaskDispatcher.java:121)
      at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canTake(ThrottleQueueTaskDispatcher.java:36)
      at hudson.model.queue.QueueTaskDispatcher.canTake(QueueTaskDispatcher.java:101)
      at hudson.model.Queue$JobOffer.canTake(Queue.java:254)
      at hudson.model.Queue.maintain(Queue.java:1032)

      • locked <0x00000006805417d8> (a hudson.model.Queue)
        at hudson.model.Queue$MaintainTask.doRun(Queue.java:1762)
        at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)

        Attachments

          Issue Links

            Activity

            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Upd. Issue affects categories only

            Show
            oleg_nenashev Oleg Nenashev added a comment - Upd. Issue affects categories only
            Hide
            marcomiller Marco Miller added a comment -

            Hi Oleg; we're interested in fixing this issue too (Ericsson).
            How is it going from your end, as we speak?
            Thx+ for letting me know! =)

            Show
            marcomiller Marco Miller added a comment - Hi Oleg; we're interested in fixing this issue too (Ericsson). How is it going from your end, as we speak? Thx+ for letting me know! =)
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Hello Marco,

            Pull request is waiting for review (it has been created about month ago).
            https://github.com/jenkinsci/throttle-concurrent-builds-plugin/pull/6
            Seems I need to send an another message to Andrew.

            The fix successfully runs on my installations, so we just need to merge it.

            Best regards,
            Oleg Nenashev

            Show
            oleg_nenashev Oleg Nenashev added a comment - Hello Marco, Pull request is waiting for review (it has been created about month ago). https://github.com/jenkinsci/throttle-concurrent-builds-plugin/pull/6 Seems I need to send an another message to Andrew. The fix successfully runs on my installations, so we just need to merge it. Best regards, Oleg Nenashev
            Hide
            jglick Jesse Glick added a comment -

            Working on a different fix.

            Show
            jglick Jesse Glick added a comment - Working on a different fix.
            Hide
            jglick Jesse Glick added a comment -

            https://github.com/jenkinsci/throttle-concurrent-builds-plugin/pull/10 offered as an alternative. (I have no installation to check it on, so relying on automated tests.)

            Show
            jglick Jesse Glick added a comment - https://github.com/jenkinsci/throttle-concurrent-builds-plugin/pull/10 offered as an alternative. (I have no installation to check it on, so relying on automated tests.)
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/java/hudson/plugins/throttleconcurrents/ThrottleJobProperty.java
            src/test/java/hudson/plugins/throttleconcurrents/ThrottleJobPropertyTest.java
            http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/9fc87ab447a60552129b4e1580e188b8a7df4aa7
            Log:
            [FIXED JENKINS-19623] Avoid getAllItems during getCategoryProjects as it checks permissions and so can be expensive, whereas QueueTaskDispatcher.canTake must be fast.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/hudson/plugins/throttleconcurrents/ThrottleJobProperty.java src/test/java/hudson/plugins/throttleconcurrents/ThrottleJobPropertyTest.java http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/9fc87ab447a60552129b4e1580e188b8a7df4aa7 Log: [FIXED JENKINS-19623] Avoid getAllItems during getCategoryProjects as it checks permissions and so can be expensive, whereas QueueTaskDispatcher.canTake must be fast.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            src/main/java/hudson/plugins/throttleconcurrents/ThrottleJobProperty.java
            src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java
            src/test/java/hudson/plugins/throttleconcurrents/ThrottleJobPropertyTest.java
            http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/18c7b4f136eed46bbeb07f9661dae9075921c43c
            Log:
            Merge pull request #10 from jglick/slow-getCategoryProjects-JENKINS-19623

            [FIXED JENKINS-19623] getCategoryProjects without permission checks

            Compare: https://github.com/jenkinsci/throttle-concurrent-builds-plugin/compare/1bc16cd4c9e4...18c7b4f136ee

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/hudson/plugins/throttleconcurrents/ThrottleJobProperty.java src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java src/test/java/hudson/plugins/throttleconcurrents/ThrottleJobPropertyTest.java http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/18c7b4f136eed46bbeb07f9661dae9075921c43c Log: Merge pull request #10 from jglick/slow-getCategoryProjects- JENKINS-19623 [FIXED JENKINS-19623] getCategoryProjects without permission checks Compare: https://github.com/jenkinsci/throttle-concurrent-builds-plugin/compare/1bc16cd4c9e4...18c7b4f136ee

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                oleg_nenashev Oleg Nenashev
              • Votes:
                1 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: