Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56273

Jenkins leaves nodes running in GCE if the termination failed

    Details

    • Similar Issues:

      Description

      I have seen nodes being left running in GCE after it has been removed from Jenkins because the termination failed. From the log:

      WARNING: Failed to terminate <node>
      java.net.SocketTimeoutException: connect timed out
              at java.net.PlainSocketImpl.socketConnect(Native Method)
              at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
              at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
              at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
              at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
              at java.net.Socket.connect(Socket.java:589)
              at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:666)
              at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
              at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
              at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
              at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
              at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
              at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
              at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
              at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
              at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
              at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:162)
              at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:93)
              at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
              at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
              at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
              at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
              at com.google.jenkins.plugins.computeengine.client.ComputeClient.getInstance(ComputeClient.java:370)
              at com.google.jenkins.plugins.computeengine.client.ComputeClient.terminateInstanceWithStatus(ComputeClient.java:361)
              at com.google.jenkins.plugins.computeengine.ComputeEngineInstance._terminate(ComputeEngineInstance.java:75)
              at hudson.slaves.AbstractCloudSlave.terminate(AbstractCloudSlave.java:67)
              at hudson.slaves.CloudRetentionStrategy.check(CloudRetentionStrategy.java:59)
              at hudson.slaves.CloudRetentionStrategy.check(CloudRetentionStrategy.java:43)
              at hudson.slaves.ComputerRetentionWork$1.run(ComputerRetentionWork.java:72)
              at hudson.model.Queue._withLock(Queue.java:1381)
              at hudson.model.Queue.withLock(Queue.java:1258)
              at hudson.slaves.ComputerRetentionWork.doRun(ComputerRetentionWork.java:63)
              at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
              at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)

      If a node can't be terminated the plugin cannot terminate a node it should retry rather than just forgetting about the node.

        Attachments

          Activity

          Hide
          ingwar Karol Lassak added a comment -

          It was fixed by cleaning "lost nodes", released in 1.0.9

          Show
          ingwar Karol Lassak added a comment - It was fixed by cleaning "lost nodes", released in 1.0.9
          Hide
          zombiemoose Rachel Yen added a comment -

          Hi James, can you verify which version of the plugin you're running?

          Show
          zombiemoose Rachel Yen added a comment - Hi James, can you verify which version of the plugin you're running?
          Hide
          organised_chaos James Robson added a comment -

          I'm not running 1.0.9, and we should have resolved the connection errors, so I think this ticket can be closed. If I do see something like this after updating the plugin I can open a new ticket.

          Show
          organised_chaos James Robson added a comment - I'm not running 1.0.9, and we should have resolved the connection errors, so I think this ticket can be closed. If I do see something like this after updating the plugin I can open a new ticket.

            People

            • Assignee:
              zombiemoose Rachel Yen
              Reporter:
              organised_chaos James Robson
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: