Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55033

Jenkins attempts to connect to Windows slave before GCP VM is provisioned

    Details

    • Similar Issues:

      Description

      I've created slave VM image with Packer and configured GCP cloud in Jenkins settings according to this guide.

      When I start job requiring "windows" agent Jenkins starts to provision agents in a loop about every minute until I notice 10+ new agents and cancel the build.

      I've attached one "iteration" of "provision loop", which seems to indicate that Jenkins doesn't honor 300 second launch timeout and prematurely gives up on agent and starts to provision next one.

        Attachments

          Activity

          Hide
          zombiemoose Rachel Yen added a comment -

          Going through the logs, this is what stands out to me:

          WARNING: Error:
          java.lang.NullPointerException
          	at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.newInstance(SSHAuthenticator.java:191)
          	at com.google.jenkins.plugins.computeengine.ComputeEngineWindowsLauncher.authenticateSSH(ComputeEngineWindowsLauncher.java:152)
          	at com.google.jenkins.plugins.computeengine.ComputeEngineWindowsLauncher.bootstrap(ComputeEngineWindowsLauncher.java:181)
          	at com.google.jenkins.plugins.computeengine.ComputeEngineWindowsLauncher.launch(ComputeEngineWindowsLauncher.java:98)
          	at com.google.jenkins.plugins.computeengine.ComputeEngineComputerLauncher.launch(ComputeEngineComputerLauncher.java:127)
          	at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294)
          	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
          	at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          	at java.lang.Thread.run(Thread.java:748)
          

          It seems we are having issues with your private key credentials. The only time I've had a similar issue to yours was during integration testing when the credentials created weren't properly registered in the system. Can you verify that you have the ssh credentials[ plugin |https://wiki.jenkins.io/display/JENKINS/SSH+Credentials+Plugin] installed and that you created credentials using this plugin? Otherwise, did you create a username and password credential? One of these must be selected under Configure. 

          Show
          zombiemoose Rachel Yen added a comment - Going through the logs, this is what stands out to me: WARNING: Error: java.lang.NullPointerException at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.newInstance(SSHAuthenticator.java:191) at com.google.jenkins.plugins.computeengine.ComputeEngineWindowsLauncher.authenticateSSH(ComputeEngineWindowsLauncher.java:152) at com.google.jenkins.plugins.computeengine.ComputeEngineWindowsLauncher.bootstrap(ComputeEngineWindowsLauncher.java:181) at com.google.jenkins.plugins.computeengine.ComputeEngineWindowsLauncher.launch(ComputeEngineWindowsLauncher.java:98) at com.google.jenkins.plugins.computeengine.ComputeEngineComputerLauncher.launch(ComputeEngineComputerLauncher.java:127) at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748) It seems we are having issues with your private key credentials. The only time I've had a similar issue to yours was during integration testing when the credentials created weren't properly registered in the system. Can you verify that you have the ssh credentials[ plugin | https://wiki.jenkins.io/display/JENKINS/SSH+Credentials+Plugin ] installed and that you created credentials using this plugin? Otherwise, did you create a username and password credential? One of these must be selected under Configure. 
          Hide
          zarbis Vadim Gusev added a comment -

          Rachel Yen I'm using login-pass pair. I can manually create VM instance and SSH into it with those credentials. I've re-checked that it's correct by re-creating it in Jenkins credential store and re-enabling it in plugin config.

          I've re-created Packer image with actual SSH key and it actually works. So it's password auth that's broken.

          Also aforementioned guide lacks Git resulting in `java.io.IOException: Cannot run program "git" (in directory "C:\JenkinsSlave\workspace\appdater_preloader_develop"): CreateProcess error=2, The system cannot find the file specified`.

          Adding `choco install -y git` to provision script did the trick.

          Show
          zarbis Vadim Gusev added a comment - Rachel Yen I'm using login-pass pair. I can manually create VM instance and SSH into it with those credentials. I've re-checked that it's correct by re-creating it in Jenkins credential store and re-enabling it in plugin config. I've re-created Packer image with actual SSH key and it actually works. So it's password auth that's broken. Also aforementioned guide lacks Git resulting in `java.io.IOException: Cannot run program "git" (in directory "C:\JenkinsSlave\workspace\appdater_preloader_develop"): CreateProcess error=2, The system cannot find the file specified`. Adding `choco install -y git` to provision script did the trick.
          Hide
          zombiemoose Rachel Yen added a comment -

          Ah, ok, I will need to look into username/password authentication. Thanks for pointing this out.

          That's very interesting, are you using the windows 2016 image? I've never needed to install git separately. That's a very interesting issue I will have to consider to better help our users.

          Show
          zombiemoose Rachel Yen added a comment - Ah, ok, I will need to look into username/password authentication. Thanks for pointing this out. That's very interesting, are you using the windows 2016 image? I've never needed to install git separately. That's a very interesting issue I will have to consider to better help our users.
          Hide
          zarbis Vadim Gusev added a comment -

          Yep, I've used exactly the same provision script as in guide. Exact version from Packer logs: googlecompute: Using image: windows-server-2016-dc-v20181113.

          Show
          zarbis Vadim Gusev added a comment - Yep, I've used exactly the same provision script as in guide. Exact version from Packer logs: googlecompute: Using image: windows-server-2016-dc-v20181113.
          Hide
          zombiemoose Rachel Yen added a comment -

          I have a commit out that will fix the password issue. Regarding the startup script to install git, I will need to look into that. Will close this when we release my fix.

          Show
          zombiemoose Rachel Yen added a comment - I have a commit out that will fix the password issue. Regarding the startup script to install git, I will need to look into that. Will close this when we release my fix.
          Hide
          zombiemoose Rachel Yen added a comment -

          Just released the fix. Let us know if you have further issues, but I will resolve this one.

          Show
          zombiemoose Rachel Yen added a comment - Just released the fix. Let us know if you have further issues, but I will resolve this one.

            People

            • Assignee:
              zombiemoose Rachel Yen
              Reporter:
              zarbis Vadim Gusev
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: