Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-58684

Linux max task resource exhaustion after excessive timer thread creation

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: sse-gateway-plugin
    • Labels:
    • Environment:
      Jenkins 2.176.2 using Ubuntu 18.04 package file from pkg.jenkins.io/debian-stable
      AMD Threadripper 32-core/64-core machine with 128GB of RAM
      Single master node with 20 executors
    • Similar Issues:

      Description

      Jenkins is exhausting resources on my machine seemingly to create 10k + timer threads. This thread creation then hits the systemd imposed 15% of task limit set by the kernel (/proc/sys/kernel/pid_max = 131072) which is 19660 by default on my system.

      This particular java heap dump shows that 4675 of the total 4877 match the expression Timer-.* and have the Context Class Loader -> org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader.

      Bottom right corner of my Jenkins UI confirms I'm running the latest version (Jenkins ver. 2.176.2)with all plugins updated on OpenJDK Runtime Environment (build 1.8.0_222-b05) with Ubuntu 18.04 as of 2019.07.23.

       

      To dig deeper, after the resource limit is hit, threads fail to create and Jenkins becomes unstable until restarted, logs contain many instances of the following:

      SEVERE: Timer task hudson.model.Queue$MaintainTask@621715d8 failed
      java.lang.OutOfMemoryError: unable to create new native thread
      

       
      NetData on the system shows that if it doesn't hit the thread limit it will spike and often recover correctly suggesting the threads are cleaned up at some point if it doesn't derail the process first.

      I've increased the systemd limit on the system and the problem hasn't derailed the jenkins process in a few days by adding the following to the systemd service file:

      TasksMax=32768
      

      I have been capturing heaps with some automation tools and catch several heaps exceeding 20k threads in the last few days signifying that the process is still occurring and likely affecting other users, they probably aren't noticing though.

       

      Not sure if this is related to a plugin (listed below with version) or Jenkins core code?

       

      [
        {
          "plugin": "Apache HttpComponents Client 4.x API Plugin (apache-httpcomponents-client-4-api)",
          "version": "4.5.5-3.0"
        },
        {
          "plugin": "Authentication Tokens API Plugin (authentication-tokens)",
          "version": "1.3"
        },
        {
          "plugin": "Authorize Project (authorize-project)",
          "version": "1.3.0"
        },
        {
          "plugin": "Autofavorite for Blue Ocean (blueocean-autofavorite)",
          "version": "1.2.4"
        },
        {
          "plugin": "Bitbucket Branch Source Plugin (cloudbees-bitbucket-branch-source)",
          "version": "2.4.5"
        },
        {
          "plugin": "Bitbucket Pipeline for Blue Ocean (blueocean-bitbucket-pipeline)",
          "version": "1.17.0"
        },
        {
          "plugin": "Bitbucket Plugin (bitbucket)",
          "version": "1.1.10"
        },
        {
          "plugin": "Blue Ocean (blueocean)",
          "version": "1.17.0"
        },
        {
          "plugin": "Blue Ocean Core JS (blueocean-core-js)",
          "version": "1.17.0"
        },
        {
          "plugin": "Blue Ocean Executor Info (blueocean-executor-info)",
          "version": "1.17.0"
        },
        {
          "plugin": "Blue Ocean Pipeline Editor (blueocean-pipeline-editor)",
          "version": "1.17.0"
        },
        {
          "plugin": "Branch API Plugin (branch-api)",
          "version": "2.5.3"
        },
        {
          "plugin": "Build Timeout (build-timeout)",
          "version": "1.19"
        },
        {
          "plugin": "Command Agent Launcher Plugin (command-launcher)",
          "version": "1.3"
        },
        {
          "plugin": "Common API for Blue Ocean (blueocean-commons)",
          "version": "1.17.0"
        },
        {
          "plugin": "Conditional BuildStep (conditional-buildstep)",
          "version": "1.3.6"
        },
        {
          "plugin": "Config API for Blue Ocean (blueocean-config)",
          "version": "1.17.0"
        },
        {
          "plugin": "Copy Artifact Plugin (copyartifact)",
          "version": "1.42.1"
        },
        {
          "plugin": "Credentials Binding Plugin (credentials-binding)",
          "version": "1.19"
        },
        {
          "plugin": "Credentials Plugin (credentials)",
          "version": "2.2.0"
        },
        {
          "plugin": "Dashboard for Blue Ocean (blueocean-dashboard)",
          "version": "1.17.0"
        },
        {
          "plugin": "Design Language (jenkins-design-language)",
          "version": "1.17.0"
        },
        {
          "plugin": "Display URL API (display-url-api)",
          "version": "2.3.1"
        },
        {
          "plugin": "Display URL for Blue Ocean (blueocean-display-url)",
          "version": "2.3.0"
        },
        {
          "plugin": "Docker Commons Plugin (docker-commons)",
          "version": "1.15"
        },
        {
          "plugin": "Docker Pipeline (docker-workflow)",
          "version": "1.18"
        },
        {
          "plugin": "Durable Task Plugin (durable-task)",
          "version": "1.30"
        },
        {
          "plugin": "Email Extension Plugin (email-ext)",
          "version": "2.66"
        },
        {
          "plugin": "Events API for Blue Ocean (blueocean-events)",
          "version": "1.17.0"
        },
        {
          "plugin": "External Monitor Job Type Plugin (external-monitor-job)",
          "version": "1.7"
        },
        {
          "plugin": "Favorite (favorite)",
          "version": "2.3.2"
        },
        {
          "plugin": "Folders Plugin (cloudbees-folder)",
          "version": "6.9"
        },
        {
          "plugin": "GIT server Plugin (git-server)",
          "version": "1.7"
        },
        {
          "plugin": "Git Pipeline for Blue Ocean (blueocean-git-pipeline)",
          "version": "1.17.0"
        },
        {
          "plugin": "Git client plugin (git-client)",
          "version": "3.0.0-rc"
        },
        {
          "plugin": "Git plugin (git)",
          "version": "4.0.0-rc"
        },
        {
          "plugin": "GitHub API Plugin (github-api)",
          "version": "1.95"
        },
        {
          "plugin": "GitHub Branch Source Plugin (github-branch-source)",
          "version": "2.5.4"
        },
        {
          "plugin": "GitHub Pipeline for Blue Ocean (blueocean-github-pipeline)",
          "version": "1.17.0"
        },
        {
          "plugin": "GitHub plugin (github)",
          "version": "1.29.4"
        },
        {
          "plugin": "Google Login Plugin (google-login)",
          "version": "1.6"
        },
        {
          "plugin": "Gradle Plugin (gradle)",
          "version": "1.33"
        },
        {
          "plugin": "HTML Publisher plugin (htmlpublisher)",
          "version": "1.18"
        },
        {
          "plugin": "HTTP Request Plugin (http_request)",
          "version": "1.8.23"
        },
        {
          "plugin": "Handy Uri Templates 2.x API Plugin (handy-uri-templates-2-api)",
          "version": "2.1.7-1.0"
        },
        {
          "plugin": "Icon Shim Plugin (icon-shim)",
          "version": "2.0.3"
        },
        {
          "plugin": "JIRA Integration for Blue Ocean (blueocean-jira)",
          "version": "1.17.0"
        },
        {
          "plugin": "JIRA plugin (jira)",
          "version": "3.0.8"
        },
        {
          "plugin": "JSch dependency plugin (jsch)",
          "version": "0.1.55"
        },
        {
          "plugin": "JUnit Plugin (junit)",
          "version": "1.28"
        },
        {
          "plugin": "JWT for Blue Ocean (blueocean-jwt)",
          "version": "1.17.0"
        },
        {
          "plugin": "Jackson 2 API Plugin (jackson2-api)",
          "version": "2.9.9.1"
        },
        {
          "plugin": "JavaScript GUI Lib: ACE Editor bundle plugin (ace-editor)",
          "version": "1.1"
        },
        {
          "plugin": "JavaScript GUI Lib: Handlebars bundle plugin (handlebars)",
          "version": "1.1.1"
        },
        {
          "plugin": "JavaScript GUI Lib: Moment.js bundle plugin (momentjs)",
          "version": "1.1.1"
        },
        {
          "plugin": "JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin (jquery-detached)",
          "version": "1.2.1"
        },
        {
          "plugin": "Javadoc Plugin (javadoc)",
          "version": "1.5"
        },
        {
          "plugin": "LDAP Plugin (ldap)",
          "version": "1.20"
        },
        {
          "plugin": "Lockable Resources plugin (lockable-resources)",
          "version": "2.5"
        },
        {
          "plugin": "Mailer Plugin (mailer)",
          "version": "1.23"
        },
        {
          "plugin": "MapDB API Plugin (mapdb-api)",
          "version": "1.0.9.0"
        },
        {
          "plugin": "Matrix Authorization Strategy Plugin (matrix-auth)",
          "version": "2.4.2"
        },
        {
          "plugin": "Matrix Project Plugin (matrix-project)",
          "version": "1.14"
        },
        {
          "plugin": "Maven Integration plugin (maven-plugin)",
          "version": "3.3"
        },
        {
          "plugin": "Mercurial plugin (mercurial)",
          "version": "2.7"
        },
        {
          "plugin": "Metrics Plugin (metrics)",
          "version": "4.0.2.5"
        },
        {
          "plugin": "OWASP Markup Formatter Plugin (antisamy-markup-formatter)",
          "version": "1.5"
        },
        {
          "plugin": "Oracle Java SE Development Kit Installer Plugin (jdk-tool)",
          "version": "1.3"
        },
        {
          "plugin": "PAM Authentication plugin (pam-auth)",
          "version": "1.5.1"
        },
        {
          "plugin": "Personalization for Blue Ocean (blueocean-personalization)",
          "version": "1.17.0"
        },
        {
          "plugin": "Pipeline (workflow-aggregator)",
          "version": "2.6"
        },
        {
          "plugin": "Pipeline Graph Analysis Plugin (pipeline-graph-analysis)",
          "version": "1.10"
        },
        {
          "plugin": "Pipeline SCM API for Blue Ocean (blueocean-pipeline-scm-api)",
          "version": "1.17.0"
        },
        {
          "plugin": "Pipeline Utility Steps (pipeline-utility-steps)",
          "version": "2.3.0"
        },
        {
          "plugin": "Pipeline implementation for Blue Ocean (blueocean-pipeline-api-impl)",
          "version": "1.17.0"
        },
        {
          "plugin": "Pipeline: API (workflow-api)",
          "version": "2.35"
        },
        {
          "plugin": "Pipeline: Basic Steps (workflow-basic-steps)",
          "version": "2.18"
        },
        {
          "plugin": "Pipeline: Build Step (pipeline-build-step)",
          "version": "2.9"
        },
        {
          "plugin": "Pipeline: Declarative (pipeline-model-definition)",
          "version": "1.3.9"
        },
        {
          "plugin": "Pipeline: Declarative Agent API (pipeline-model-declarative-agent)",
          "version": "1.1.1"
        },
        {
          "plugin": "Pipeline: Declarative Extension Points API (pipeline-model-extensions)",
          "version": "1.3.9"
        },
        {
          "plugin": "Pipeline: GitHub Groovy Libraries (pipeline-github-lib)",
          "version": "1.0"
        },
        {
          "plugin": "Pipeline: Groovy (workflow-cps)",
          "version": "2.72"
        },
        {
          "plugin": "Pipeline: Input Step (pipeline-input-step)",
          "version": "2.10"
        },
        {
          "plugin": "Pipeline: Job (workflow-job)",
          "version": "2.33"
        },
        {
          "plugin": "Pipeline: Milestone Step (pipeline-milestone-step)",
          "version": "1.3.1"
        },
        {
          "plugin": "Pipeline: Model API (pipeline-model-api)",
          "version": "1.3.9"
        },
        {
          "plugin": "Pipeline: Multibranch (workflow-multibranch)",
          "version": "2.21"
        },
        {
          "plugin": "Pipeline: Nodes and Processes (workflow-durable-task-step)",
          "version": "2.32"
        },
        {
          "plugin": "Pipeline: REST API Plugin (pipeline-rest-api)",
          "version": "2.11"
        },
        {
          "plugin": "Pipeline: SCM Step (workflow-scm-step)",
          "version": "2.9"
        },
        {
          "plugin": "Pipeline: Shared Groovy Libraries (workflow-cps-global-lib)",
          "version": "2.14"
        },
        {
          "plugin": "Pipeline: Stage Step (pipeline-stage-step)",
          "version": "2.3"
        },
        {
          "plugin": "Pipeline: Stage Tags Metadata (pipeline-stage-tags-metadata)",
          "version": "1.3.9"
        },
        {
          "plugin": "Pipeline: Stage View Plugin (pipeline-stage-view)",
          "version": "2.11"
        },
        {
          "plugin": "Pipeline: Step API (workflow-step-api)",
          "version": "2.20"
        },
        {
          "plugin": "Pipeline: Supporting APIs (workflow-support)",
          "version": "3.3"
        },
        {
          "plugin": "Plain Credentials Plugin (plain-credentials)",
          "version": "1.5"
        },
        {
          "plugin": "Pub-Sub \"light\" Bus (pubsub-light)",
          "version": "1.12"
        },
        {
          "plugin": "REST API for Blue Ocean (blueocean-rest)",
          "version": "1.17.0"
        },
        {
          "plugin": "REST Implementation for Blue Ocean (blueocean-rest-impl)",
          "version": "1.17.0"
        },
        {
          "plugin": "Resource Disposer Plugin (resource-disposer)",
          "version": "0.13"
        },
        {
          "plugin": "Run Condition Plugin (run-condition)",
          "version": "1.2"
        },
        {
          "plugin": "SCM API Plugin (scm-api)",
          "version": "2.6.3"
        },
        {
          "plugin": "SSH Agent Plugin (ssh-agent)",
          "version": "1.17"
        },
        {
          "plugin": "SSH Credentials Plugin (ssh-credentials)",
          "version": "1.17.1"
        },
        {
          "plugin": "SSH Slaves plugin (ssh-slaves)",
          "version": "1.30.1"
        },
        {
          "plugin": "Script Security Plugin (script-security)",
          "version": "1.61"
        },
        {
          "plugin": "Server Sent Events (SSE) Gateway Plugin (sse-gateway)",
          "version": "1.18"
        },
        {
          "plugin": "Skip Notifications Trait plugin (skip-notifications-trait)",
          "version": "1.0.3"
        },
        {
          "plugin": "Structs Plugin (structs)",
          "version": "1.19"
        },
        {
          "plugin": "Subversion Plug-in (subversion)",
          "version": "2.12.2"
        },
        {
          "plugin": "Timestamper (timestamper)",
          "version": "1.10"
        },
        {
          "plugin": "Token Macro Plugin (token-macro)",
          "version": "2.8"
        },
        {
          "plugin": "Trilead API Plugin (trilead-api)",
          "version": "1.0.3"
        },
        {
          "plugin": "Variant Plugin (variant)",
          "version": "1.2"
        },
        {
          "plugin": "WMI Windows Agents Plugin (windows-slaves)",
          "version": "1.4"
        },
        {
          "plugin": "Web for Blue Ocean (blueocean-web)",
          "version": "1.17.0"
        },
        {
          "plugin": "Workspace Cleanup Plugin (ws-cleanup)",
          "version": "0.37"
        },
        {
          "plugin": "bouncycastle API Plugin (bouncycastle-api)",
          "version": "2.17"
        },
        {
          "plugin": "i18n for Blue Ocean (blueocean-i18n)",
          "version": "1.17.0"
        }
      ]
      

        Attachments

          Issue Links

            Activity

            Hide
            efroemling Eric Froemling added a comment -

            Just chiming in here:

            I've been running into this issue for a few weeks on my old Mac I've got set up as a CI server; at some point after using the Blue Ocean UI I'd start seeing the thread count on Jenkins rise rapidly from its usual ~70ish into the thousands and then I'd start getting 'could not create native thread' errors and other strange side-effects.  I had been avoiding Blue Ocean for the last week or so due to this.

            I went ahead and tried the sse-gateway 1.20-SNAPSHOT posted yesterday and went back to using Blue Ocean and am happy to report I've not observed the thread leak since.  I'll keep my eye on it but so far things are looking good.  Thanks everyone who has been looking into this!

            Show
            efroemling Eric Froemling added a comment - Just chiming in here: I've been running into this issue for a few weeks on my old Mac I've got set up as a CI server; at some point after using the Blue Ocean UI I'd start seeing the thread count on Jenkins rise rapidly from its usual ~70ish into the thousands and then I'd start getting 'could not create native thread' errors and other strange side-effects.  I had been avoiding Blue Ocean for the last week or so due to this. I went ahead and tried the sse-gateway 1.20-SNAPSHOT posted yesterday and went back to using Blue Ocean and am happy to report I've not observed the thread leak since.  I'll keep my eye on it but so far things are looking good.  Thanks everyone who has been looking into this!
            Hide
            olamy Olivier Lamy added a comment -

            1.20 version has been released

            Show
            olamy Olivier Lamy added a comment - 1.20 version has been released
            Hide
            reinholdfuereder Reinhold Füreder added a comment -

            Olivier Lamy Sorry for my delayed feedback: AFAIK (no user complaints in my company – at least none that I know of...) the snapshot build of sse-gateway worked fine; and on Jenkins server neither remaining "EventDispatcher.retryProcessor" threads, nor thousands of them at the same time. => Looks good

            Show
            reinholdfuereder Reinhold Füreder added a comment - Olivier Lamy Sorry for my delayed feedback: AFAIK (no user complaints in my company – at least none that I know of...) the snapshot build of sse-gateway worked fine; and on Jenkins server neither remaining "EventDispatcher.retryProcessor" threads, nor thousands of them at the same time. => Looks good
            Hide
            2bluesc Kyle Manna added a comment -

            Updated on my server, thanks for all the hard work!

            Show
            2bluesc Kyle Manna added a comment - Updated on my server, thanks for all the hard work!
            Hide
            sorenfriis Søren Friis added a comment -

            There seems to be a problem with Showing Blue Ocean in MS Edge after this update.
            https://issues.jenkins-ci.org/browse/JENKINS-59291

             

            Show
            sorenfriis Søren Friis added a comment - There seems to be a problem with Showing Blue Ocean in MS Edge after this update. https://issues.jenkins-ci.org/browse/JENKINS-59291  

              People

              • Assignee:
                olamy Olivier Lamy
                Reporter:
                2bluesc Kyle Manna
              • Votes:
                9 Vote for this issue
                Watchers:
                16 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: