Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-23904

SlowRequestCheck= kills UI performance and pegs CPU at 100%

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: support-core-plugin
    • Labels:
      None
    • Environment:
      2 * E5-2690 (16 real cores, 32 inc hyperthreading)
      112 MB RAM
      -Xmx24g -Xms16g
    • Similar Issues:

      Description

      The slow request checker pegs the JDK VM_Thread at 100% cpu as it is in a constant loop grabing thread dumps.

      Getting full thread dumps on a largish system (18 slaves, 5 polls, several web requests) every 3 seconds is not a great idea.

      There is no way to tune this and the only way to fix this is to uninstall the support core plugin and restart jenkins.

      depending on the JDK you may be able to terminate the Slow request thread with the following

      import jenkins.util.Timer 
      import java.util.concurrent.*
      
      def timer = Timer.get() 
      def tasks = timer.queue.toArray()
      
      def field = FutureTask.class.getDeclaredField("callable") 
      field.setAccessible(true)
      
      for (task in tasks) { 
        if (field.get(task).task.class == com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker.class) { 
          println(task.toString() + " <-----========== SlowRequestChecker") 
          println(task.cancel(true)) 
          println(timer.queue.remove()) 
        } 
      }
      

      Once the requestChecker is removed the UI response is near instantanious.

        Attachments

          Activity

          Hide
          kohsuke Kohsuke Kawaguchi added a comment -

          I see. We'll split this off to a separate plugin so that it can be removed entirely.

          I wonder using ThreadMXBean.getThreadInfo() that specifies thread IDs explicitly would reduce the overhead.

          Show
          kohsuke Kohsuke Kawaguchi added a comment - I see. We'll split this off to a separate plugin so that it can be removed entirely. I wonder using ThreadMXBean.getThreadInfo() that specifies thread IDs explicitly would reduce the overhead.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Stephen Connolly
          Path:
          src/main/java/com/cloudbees/jenkins/support/slowrequest/SlowRequestChecker.java
          http://jenkins-ci.org/commit/support-core-plugin/1499fe2b79a556e7988801c792c49fc15d379d58
          Log:
          [FIXED JENKINS-23904] SlowRequestCheck= kills UI performance and pegs CPU at 100%

          • Made the timings for this configurable by a system property
          • Only collect thread dump if there is a slow request
          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: src/main/java/com/cloudbees/jenkins/support/slowrequest/SlowRequestChecker.java http://jenkins-ci.org/commit/support-core-plugin/1499fe2b79a556e7988801c792c49fc15d379d58 Log: [FIXED JENKINS-23904] SlowRequestCheck= kills UI performance and pegs CPU at 100% Made the timings for this configurable by a system property Only collect thread dump if there is a slow request
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Stephen Connolly
          Path:
          src/main/java/com/cloudbees/jenkins/support/slowrequest/SlowRequestChecker.java
          src/main/java/com/cloudbees/jenkins/support/slowrequest/SlowRequestFilter.java
          http://jenkins-ci.org/commit/support-core-plugin/8c8df7ae0977af0f31324906d4f43764f03846fc
          Log:
          [FIXED JENKINS-23904] SlowRequestCheck= kills UI performance and pegs CPU at 100%

          • Provide a means to completely disable the slow request checker both at startup and in a running Jenkins

          Compare: https://github.com/jenkinsci/support-core-plugin/compare/2b43d350f9a3...8c8df7ae0977

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: src/main/java/com/cloudbees/jenkins/support/slowrequest/SlowRequestChecker.java src/main/java/com/cloudbees/jenkins/support/slowrequest/SlowRequestFilter.java http://jenkins-ci.org/commit/support-core-plugin/8c8df7ae0977af0f31324906d4f43764f03846fc Log: [FIXED JENKINS-23904] SlowRequestCheck= kills UI performance and pegs CPU at 100% Provide a means to completely disable the slow request checker both at startup and in a running Jenkins Compare: https://github.com/jenkinsci/support-core-plugin/compare/2b43d350f9a3...8c8df7ae0977
          Hide
          stephenconnolly Stephen Connolly added a comment -

          @Kohsuke It would seem our paths crossed... If you want to try the thread ID specific version too that could help further

          Show
          stephenconnolly Stephen Connolly added a comment - @Kohsuke It would seem our paths crossed... If you want to try the thread ID specific version too that could help further
          Hide
          stephenconnolly Stephen Connolly added a comment -

          FYI the root cause of the issue as I see it is that the thread dump was eager not lazy... you had a check to lazy load the thread dump but you were eagerly fetching it up front. I removed the eager fetch in 1499fe2b79a556e7988801c792c49fc15d379d58

          I am unsure that we need to spin off to a separate plugin.

          Show
          stephenconnolly Stephen Connolly added a comment - FYI the root cause of the issue as I see it is that the thread dump was eager not lazy... you had a check to lazy load the thread dump but you were eagerly fetching it up front. I removed the eager fetch in 1499fe2b79a556e7988801c792c49fc15d379d58 I am unsure that we need to spin off to a separate plugin.
          Hide
          stephenconnolly Stephen Connolly added a comment -

          https://github.com/jenkinsci/support-core-plugin/tree/slow-request-alt-strategy is an additional change that could perhaps help.

          I also note that deadlock analysis etc is missing if you use the thread specific getThreadInfo() so I am unsure if that is a good idea or not to drop from the slow request checker

          Show
          stephenconnolly Stephen Connolly added a comment - https://github.com/jenkinsci/support-core-plugin/tree/slow-request-alt-strategy is an additional change that could perhaps help. I also note that deadlock analysis etc is missing if you use the thread specific getThreadInfo() so I am unsure if that is a good idea or not to drop from the slow request checker

            People

            • Assignee:
              Unassigned
              Reporter:
              teilo James Nord
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: