Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50984

SSHLauncher/Fingerprint Thread Locking Stopping Dynamic Slave Launch

    Details

    • Similar Issues:

      Description

      Curious if someone can help me unpack this.  We recently upgraded Jenkins.  We use the Docker-plugin to dynamically provision slaves and we're now running into a situation where the slaves do not properly finish provisioning (The SSH connection is never established). When taking a thread dump there is a very large number of Blocked threads on the SSHLauncher teardown and Fingerprinting for some reason, here's the dumps:

      "Computer.threadPoolForRemoting [#1020]" daemon prio=5 BLOCKED
      	hudson.plugins.sshslaves.SSHLauncher.tearDownConnection(SSHLauncher.java:1407)
      	hudson.plugins.sshslaves.SSHLauncher.afterDisconnect(SSHLauncher.java:1403)
      	com.nirima.jenkins.plugins.docker.launcher.DockerComputerLauncher.afterDisconnect(DockerComputerLauncher.java:71)
      	hudson.slaves.SlaveComputer$3.run(SlaveComputer.java:665)
      	jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      	java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	java.lang.Thread.run(Thread.java:748)
      
      "Computer.threadPoolForRemoting [#1019]" daemon prio=5 BLOCKED
      	hudson.model.Fingerprint.save(Fingerprint.java:1238)
      	hudson.BulkChange.commit(BulkChange.java:98)
      	com.cloudbees.plugins.credentials.CredentialsProvider.trackAll(CredentialsProvider.java:1533)
      	com.cloudbees.plugins.credentials.CredentialsProvider.track(CredentialsProvider.java:1478)
      	hudson.plugins.sshslaves.SSHLauncher.launch(SSHLauncher.java:866)
      	com.nirima.jenkins.plugins.docker.launcher.DockerComputerLauncher.launch(DockerComputerLauncher.java:66)
      	hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:288)
      	jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
      	java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	java.lang.Thread.run(Thread.java:748)
      

      The thread dump has many of these (when things get bad it gets to 100's).  We're currently planning a Docker-plugin upgrade and moving away from the SSH Launcher but I'm looking for ideas as to why this may be happening.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ifernandezcalvo Ivan Fernandez Calvo
                Reporter:
                maxfields2000 Maxfield Stewart
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: