Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-20046

Massive Jenkins slowdown when jobs in Queue (due to Queue.maintain())

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: core
    • Labels:
    • Environment:
      Ubuntu 12.04
      Jenkins 1.509.3
      Up-to-date plugins
    • Similar Issues:

      Description

      As soon as more than a handful builds get queued, the entire GUI crawls to a halt.

      The reason is that the executor thread running the "Queue.maintain()" method is holding the exclusive lock on the queue, but starts a very time-consuming loop on creating the list of applicable hosts matching a certain label.

      Due to this, every Jenkins GUI page and every method that needs access to the Queue gets delayed by ~30 seconds; with the delay rising the more builds are in the queue, due to Queue.maintain() being called more often.

      The server only becomes responsive again, once the entire queue is empty. Setting the server to "shutdown now" does not help.

      A usual stack trace when this occurs looks like this (first from /threadDump; the second from jstack during a different time):
      {{
      "Executor #6 for musxbird038" prio=10 tid=0x00007fe108024800 nid=0x7008 runnable [0x00007fe0f5a99000]
      java.lang.Thread.State: RUNNABLE
      at hudson.model.Slave.getLabelString(Slave.java:245)
      at hudson.model.Node.getAssignedLabels(Node.java:241)
      at hudson.model.Label.matches(Label.java:168)
      at hudson.model.Label.getNodes(Label.java:193)
      at hudson.model.Label.contains(Label.java:405)
      at hudson.model.Node.canTake(Node.java:322)
      at hudson.model.Queue$JobOffer.canTake(Queue.java:250)
      at hudson.model.Queue.maintain(Queue.java:1032)

      • locked <0x00000000e01d3490> (a hudson.model.Queue)
        at hudson.model.Queue.pop(Queue.java:863)
      • locked <0x00000000e01d3490> (a hudson.model.Queue)
        at hudson.model.Executor.grabJob(Executor.java:285)
        at hudson.model.Executor.run(Executor.java:206)
      • locked <0x00000000e01d3490> (a hudson.model.Queue)

      "Executor #0 for musxbird006" Id=591 Group=main RUNNABLE
      at java.util.TreeMap.successor(TreeMap.java:1975)
      at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1101)
      at java.util.TreeMap$KeyIterator.next(TreeMap.java:1154)
      at java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1010)
      at hudson.model.Label$2.resolve(Label.java:159)
      at hudson.model.Label$2.resolve(Label.java:157)
      at hudson.model.labels.LabelAtom.matches(LabelAtom.java:149)
      at hudson.model.labels.LabelExpression$Binary.matches(LabelExpression.java:124)
      at hudson.model.Label.matches(Label.java:157)
      at hudson.model.Label.matches(Label.java:168)
      at hudson.model.Label.getNodes(Label.java:193)
      at hudson.model.Label.contains(Label.java:405)
      at hudson.model.Node.canTake(Node.java:322)
      at hudson.model.Queue$JobOffer.canTake(Queue.java:250)
      at hudson.model.Queue.maintain(Queue.java:1032)

      • locked hudson.model.Queue@2962c1e0
        at hudson.model.Queue.pop(Queue.java:863)
      • locked hudson.model.Queue@2962c1e0
        at hudson.model.Executor.grabJob(Executor.java:285)
        at hudson.model.Executor.run(Executor.java:206)
      • locked hudson.model.Queue@2962c1e0
        }}

      As you can see, the Queue.maintain() method does finish successfully, but needs more than 30 seconds for it. The server does not stop working and will return to normal once the queue has been fully processed.

      We have ~20 nodes with 12 executor slots each running (= 240 executor threads). There is an equal amount of jobs running, but not all of them consume CPU time on the host (most are idling and waiting for certain events).

      This issue has occurred since upgrading from 1.509.1 to 1.509.3.

      Thanks in advance.

        Attachments

          Issue Links

            Activity

            Hide
            jglick Jesse Glick added a comment -

            Florian Manschwetus your issue is totally unrelated: JENKINS-40934

            Show
            jglick Jesse Glick added a comment - Florian Manschwetus your issue is totally unrelated:  JENKINS-40934
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            I doubt it makes sense to keep this issue open. The queue has significantly changed in 1.609.x, and there are other performance tweaks like the recent https://github.com/jenkinsci/jenkins/pull/3038/files Alexander A. The inputs here are probably not relevant anymore

            I would just close this issue as Resolved

            Show
            oleg_nenashev Oleg Nenashev added a comment - I doubt it makes sense to keep this issue open. The queue has significantly changed in 1.609.x, and there are other performance tweaks like the recent https://github.com/jenkinsci/jenkins/pull/3038/files Alexander A . The inputs here are probably not relevant anymore I would just close this issue as Resolved
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Akbashev Alexander
            Path:
            core/src/main/java/hudson/model/Queue.java
            test/src/test/java/hudson/model/QueueTest.java
            http://jenkins-ci.org/commit/jenkins/be0238644911948da4123b5338f0299198dcc048
            Log:
            JENKINS-20046 - Do not query queue dispatchers from UI (#3038)

            • Do not query queue dispatchers from UI
            • Address comments from review
            • Restore old constructors and mark them as @deprecated
            • Optimise query from UI even more
            • Check non-concurrent builds in getCauseOfBlockageForItem
            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Akbashev Alexander Path: core/src/main/java/hudson/model/Queue.java test/src/test/java/hudson/model/QueueTest.java http://jenkins-ci.org/commit/jenkins/be0238644911948da4123b5338f0299198dcc048 Log: JENKINS-20046 - Do not query queue dispatchers from UI (#3038) Do not query queue dispatchers from UI Address comments from review Restore old constructors and mark them as @deprecated Optimise query from UI even more Check non-concurrent builds in getCauseOfBlockageForItem
            Hide
            danielbeck Daniel Beck added a comment -

            Does Alexander A's change towards 2.85 resolve this issue?

            Show
            danielbeck Daniel Beck added a comment - Does Alexander A 's change towards 2.85 resolve this issue?
            Hide
            flominatortm Florian Straub added a comment - - edited

            We still have this behavior running v2.92 ...

            Update: ... but actually we managed to fix it by enabling LDAP caching in "global security"!

            Show
            flominatortm Florian Straub added a comment - - edited We still have this behavior running v2.92 ... Update: ... but actually we managed to fix it by enabling LDAP caching in "global security"!

              People

              • Assignee:
                Unassigned
                Reporter:
                mhschroe Martin Schröder
              • Votes:
                13 Vote for this issue
                Watchers:
                21 Start watching this issue

                Dates

                • Created:
                  Updated: