Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-27041

Thousands of threads created by thousands of GitHub pushes

XMLWordPrintable

      A Jenkins instance using this plugin was found to have more than 4400 threads with this stack trace:

      "Waiting to acquire /…/workspace/… : Computer.threadPoolForRemoting [#37]" daemon prio=10 tid=0x… nid=0x… in Object.wait() [0x…]
          java.lang.Thread.State: WAITING (on object monitor)
             at java.lang.Object.wait(Native Method)
             at java.lang.Object.wait(Object.java:503)
             at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:245)
             - locked <0x…> (a hudson.slaves.WorkspaceList)
             at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:224)
             - locked <0x…> (a hudson.slaves.WorkspaceList)
             at hudson.model.AbstractProject.pollWithWorkspace(AbstractProject.java:1446)
             at hudson.model.AbstractProject._poll(AbstractProject.java:1425)
             at hudson.model.AbstractProject.poll(AbstractProject.java:1336)
             at com.cloudbees.jenkins.GitHubPushTrigger$1.runPolling(GitHubPushTrigger.java:81)
             at com.cloudbees.jenkins.GitHubPushTrigger$1.run(GitHubPushTrigger.java:106)
             at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
             at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
             at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
             at java.util.concurrent.FutureTask.run(FutureTask.java:262)
             at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
             at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
             at java.lang.Thread.run(Thread.java:745)
      

      Looking at the logs, there were 1150 GitHub push notifications for this job over 12 hours. Each created a polling thread and competed for a lock on the workspace. Each polling request took 1-6 seconds.

      Such a high thread count had other knock on effects, such as high CPU usage when counting threads.

      So we should figure out a way to either eliminate the duplicate polling or just keep the thread count down.

            recampbell Ryan Campbell
            recampbell Ryan Campbell
            Votes:
            3 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: