Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-42676

Unable to ssh provisionned node on openstack (rackspace) after 2.8 -> 2.14 upgrade

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Component/s: jclouds-plugin
    • Labels:
      None
    • Environment:
      java version "1.7.0_121"
      OpenJDK Runtime Environment (IcedTea 2.6.8) (7u121-2.6.8-1ubuntu0.12.04.3)
      OpenJDK 64-Bit Server VM (build 24.121-b00, mixed mode)
      Ubuntu 12.04
      jclouds-plugin 2.14
      jenkins 2.32.3
    • Similar Issues:

      Description

      After upgrade from 2.8.1- to 2.14 I can no longer provision nodes on rackspace.

      First, I had to remove "Assign Public IP" (previously on) otherwise I got "java.lang.IllegalArgumentException: Floating IPs are required by options, but the extension is not available! options:"

      Then, even after having remove that option, I cannot find a combination of credentials that allows jenkins to ssh the server to run the init script.

      The connection to rackspace does work (tested from the interface) and the hardware & image are correct because I can see in my console that the server is being provisioned.

      But once it's up, jenkins cannot SSH:

      INFO: Provisioning new jclouds node
      Mar 10, 2017 5:22:02 PM jenkins.plugins.jclouds.compute.JCloudsSlaveTemplate get
      INFO: Setting osFamily to ubuntu
      Mar 10, 2017 5:22:02 PM jenkins.plugins.jclouds.compute.JCloudsSlaveTemplate get
      INFO: Setting osVersion to 14.04
      Mar 10, 2017 5:22:02 PM jenkins.plugins.jclouds.compute.JCloudsSlaveTemplate get
      INFO: Setting hardware Id to IAD/2
      Mar 10, 2017 5:22:02 PM org.jclouds.logging.jdk.JDKLogger logInfo
      INFO: >> searching params({osFamily=ubuntu, osVersion=14.04, hardwareId=IAD/2})
      Mar 10, 2017 5:22:02 PM org.jclouds.logging.jdk.JDKLogger logInfo
      INFO: <<   matched image(IAD/9ab07706-4b9f-41ad-a8f5-8b4fda2e40f7) hardware(IAD/2) location(IAD)
      Mar 10, 2017 5:22:02 PM jenkins.plugins.jclouds.compute.JCloudsSlaveTemplate get
      INFO: Setting adminCredentialsId to 502a5302-fb98-422a-a1dd-c8998d8c5e83
      Mar 10, 2017 5:22:02 PM jenkins.plugins.jclouds.compute.JCloudsSlaveTemplate get
      INFO: Using username/privatekey as adminCredentials
      Mar 10, 2017 5:22:02 PM org.jclouds.logging.jdk.JDKLogger logInfo
      INFO: >> running 1 node group(stuff) location(IAD) image(IAD/9ab07706-4b9f-41ad-a8f5-8b4fda2e40f7) hardwareProfile(IAD/2) options({loginUser=root, loginPrivateKeyPresent=true, scriptPresent=true, userMetadata={Name=stuff}, autoAssignFloati
      ngIp=false, configDrive=false})
      Mar 10, 2017 5:22:03 PM org.jclouds.logging.jdk.JDKLogger logInfo
      INFO: >> adding node location(IAD) name(stuff-1be) image(9ab07706-4b9f-41ad-a8f5-8b4fda2e40f7) hardware(2)
      Mar 10, 2017 5:22:03 PM org.jclouds.logging.jdk.JDKLogger logInfo
      INFO: >> creating new server region(IAD) name(stuff-1be) image(9ab07706-4b9f-41ad-a8f5-8b4fda2e40f7) flavor(2) options(CreateServerOptions{keyName=null, metadata={Name=stuff, jclouds-group=stuff}, userData=null, availabilityZone=null, conf
      igDrive=false})
      Mar 10, 2017 5:22:18 PM org.jclouds.logging.jdk.JDKLogger logInfo
      INFO: << PENDING node(IAD/a8ce5366-9635-40da-a250-ec0e8df597ff)
      Mar 10, 2017 5:30:15 PM com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.actions.manual.ManualTriggerAction getServerConfig
      SEVERE: Could not find server null
      Mar 10, 2017 5:31:20 PM org.jclouds.logging.jdk.JDKLogger logInfo
      INFO: >> blocking on sockets [104.130.231.245:22, 10.176.7.13:22] for 600000 MILLISECONDS
      Mar 10, 2017 5:31:34 PM org.jclouds.logging.jdk.JDKLogger logInfo
      INFO: << socket 10.176.7.13:22 opened
      Mar 10, 2017 5:31:34 PM net.schmizz.sshj.transport.TransportImpl init
      INFO: Client identity string: SSH-2.0-SSHJ_0_9_2
      Mar 10, 2017 5:31:34 PM net.schmizz.sshj.transport.TransportImpl init
      INFO: Server identity string: SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7
      Mar 10, 2017 5:31:34 PM net.schmizz.sshj.transport.TransportImpl$1 notifyDisconnect
      INFO: Disconnected - BY_APPLICATION
      Mar 10, 2017 5:31:34 PM org.jclouds.logging.jdk.JDKLogger logInfo
      INFO: << (root:rsa[fingerprint(e1:b3:f7:6b:c5:fd:83:6c:e9:46:a4:89:90:05:24:a0),sha1(e7:de:3c:34:a4:cb:7e:8b:50:95:4d:78:93:fa:9f:18:67:7d:b2:c1)]@10.176.7.13:22) error acquiring {hostAndPort=10.176.7.13:22, loginUser=root, ssh=null, conne
      ctTimeout=60000, sessionTimeout=60000} (attempt 1 of 7): Exhausted available authentication methods
      Mar 10, 2017 5:31:34 PM net.schmizz.sshj.transport.TransportImpl init
      INFO: Client identity string: SSH-2.0-SSHJ_0_9_2
      Mar 10, 2017 5:31:34 PM net.schmizz.sshj.transport.TransportImpl init
      INFO: Server identity string: SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7
      Mar 10, 2017 5:31:34 PM net.schmizz.sshj.transport.TransportImpl$1 notifyDisconnect
      INFO: Disconnected - BY_APPLICATION
      Mar 10, 2017 5:31:34 PM org.jclouds.logging.jdk.JDKLogger logInfo
      INFO: << (root:rsa[fingerprint(e1:b3:f7:6b:c5:fd:83:6c:e9:46:a4:89:90:05:24:a0),sha1(e7:de:3c:34:a4:cb:7e:8b:50:95:4d:78:93:fa:9f:18:67:7d:b2:c1)]@10.176.7.13:22) error acquiring {hostAndPort=10.176.7.13:22, loginUser=root, ssh=null, connectTimeout=60000, sessionTimeout=60000} (attempt 2 of 7): Exhausted available authentication methods
      Mar 10, 2017 5:31:38 PM net.schmizz.sshj.transport.TransportImpl init
      INFO: Client identity string: SSH-2.0-SSHJ_0_9_2
      Mar 10, 2017 5:31:38 PM net.schmizz.sshj.transport.TransportImpl init
      INFO: Server identity string: SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7
      Mar 10, 2017 5:31:38 PM net.schmizz.sshj.transport.TransportImpl$1 notifyDisconnect
      INFO: Disconnected - BY_APPLICATION

      That ends up to

      Mar 10, 2017 5:32:14 PM org.jclouds.logging.jdk.JDKLogger logError
      SEVERE: << (root:rsa[fingerprint(e1:b3:f7:6b:c5:fd:83:6c:e9:46:a4:89:90:05:24:a0),sha1(e7:de:3c:34:a4:cb:7e:8b:50:95:4d:78:93:fa:9f:18:67:7d:b2:c1)]@10.176.7.13:22) error acquiring {hostAndPort=10.176.7.13:22, loginUser=root, ssh=null, connectTimeout=60000, sessionTimeout=60000} (out of retries - max 7): Exhausted available authentication methods
      net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods

      The documentation is not clear on what is expected in term of credentials between

      • Cloud RSA (used to be private & public key, now we need to setup a username: which one ?)
      • Jenkins Credentials
      • Admin credentials

      Which one is used to run initscript and what should I set in term of usernames ?

        Attachments

          Activity

          vaceletm Manuel Vacelet created issue -
          Hide
          felfert Fritz Elfert added a comment -

          => Regarding the "Assign Public IP" problem, please file a separate bug, ideally with an exception stacktrace as well.

          It should all have been automatically migrated. Didn't it do that?

          • The former cloud RSA public/private key is now an SSH-Credential where the username is actually irrelevant. Furthermore, only the private key is stored, because the public key is deriveable at any time from the private key - Obviously that works, otherwise it would not even attempt to start provisioning.
          • The admin credentials are used for provisioning resp. running the init-script and can be either SSH-credentials or Username/Password credentials (depends on Provider, I don't have a rackspace account so I don't know the specifics there)
          • The jenkins credentials are used for the final SSH connection that provides the actual jenkins slave connection (which eventually pushes a JRE if it does not find one). Like with the admin credentials, those can be either UsernamePassword or SSH-credentials.
          • If the checkbox "Use Pre-existing Jenkins User" is unchecked, then in addition to the initscript, a jclouds AdminAccess script is run which provisions the jenkins user.

          BUT

          In my experience with OpenStack, GoogleCompute and DigitalOcean, I found all this jclouds-builtin stuff rather crude and unstable (Depending on amount of stuff done in the init-script, it times out very often) and I never use it myself anymore. Instead use cloud-init almost everywhere (especially as i usually is already available on most preconfigured images from those providers). With cloud-init this almost never happens especially when enabling it's phone-home webhook. Have a look here:

          https://developer.rackspace.com/blog/using-cloud-init-with-rackspace-cloud/

          With cloud-init, you usually don't set an admin credential, don't use an initScript and enable "Use Pre-existing Jenkins User". Then in the cloud-init yaml you create the jenkins-user (as well as install any required packages) and then in the jenkins credentials you simply use the jenkins user previously created with cloud-init.

          Show
          felfert Fritz Elfert added a comment - => Regarding the "Assign Public IP" problem, please file a separate bug, ideally with an exception stacktrace as well. It should all have been automatically migrated. Didn't it do that? The former cloud RSA public/private key is now an SSH-Credential where the username is actually irrelevant. Furthermore, only the private key is stored, because the public key is deriveable at any time from the private key - Obviously that works, otherwise it would not even attempt to start provisioning. The admin credentials are used for provisioning resp. running the init-script and can be either SSH-credentials or Username/Password credentials (depends on Provider, I don't have a rackspace account so I don't know the specifics there) The jenkins credentials are used for the final SSH connection that provides the actual jenkins slave connection (which eventually pushes a JRE if it does not find one). Like with the admin credentials, those can be either UsernamePassword or SSH-credentials. If the checkbox "Use Pre-existing Jenkins User" is unchecked , then in addition to the initscript, a jclouds AdminAccess script is run which provisions the jenkins user. BUT In my experience with OpenStack, GoogleCompute and DigitalOcean, I found all this jclouds-builtin stuff rather crude and unstable (Depending on amount of stuff done in the init-script, it times out very often) and I never use it myself anymore. Instead use cloud-init almost everywhere (especially as i usually is already available on most preconfigured images from those providers). With cloud-init this almost never happens especially when enabling it's phone-home webhook. Have a look here: https://developer.rackspace.com/blog/using-cloud-init-with-rackspace-cloud/ With cloud-init, you usually don't set an admin credential, don't use an initScript and enable "Use Pre-existing Jenkins User". Then in the cloud-init yaml you create the jenkins-user (as well as install any required packages) and then in the jenkins credentials you simply use the jenkins user previously created with cloud-init.
          Hide
          vaceletm Manuel Vacelet added a comment -

          Thanks for the follow-up

          It should all have been automatically migrated. Didn't it do that?

          Sort of, the elements related to rackspace APIs seems to have been properly migrated (as I can see server beeing created) but for the rest it's unclear.

          The admin credentials are used for provisioning resp. running the init-script and can be either SSH-credentials or Username/Password credentials (depends on Provider, I don't have a rackspace account so I don't know the specifics there)

          Maybe there is an issue there because I either tried to:

          • Leave it blank (as it was before upgrade)
          • Set root + private key

          And none of these worked.

          So I gave a try to cloud-init, it seems to work better (at least the node is not terminated because jenkins cannot ssh it) but I didn't manage to got it right. Here was I now have in the logs:

          Mar 10, 2017 9:08:11 PM org.jclouds.logging.jdk.JDKLogger logInfo
          INFO: >> creating new server region(IAD) name(stuff-ba2) image(9ab07706-4b9f-41ad-a8f5-8b4fda2e40f7) flavor(general1-1) options(CreateServerOptions{keyName=null, metadata={Name=stuff, jclouds-group=stuff}, userData=#cloud-config
          users:
          - name: jenkins
          {{ homedir: /jenkins}}
          {{ ssh-authorized-keys:}}
          {{ - ssh-rsa AAAAB....}}
          {{ sudo: ['ALL=(ALL) NOPASSWD:ALL']}}
          {{ groups: sudo}}
          {{ shell: /bin/bash}}

          phone_home:
          {{ url: https://ci.tuleap.org/jenkins/jclouds-phonehome/}}
          {{ tries: 3}}
          {{ , availabilityZone=null, configDrive=false})}}
          Mar 10, 2017 9:08:14 PM org.jclouds.logging.jdk.JDKLogger logInfo
          INFO: << PENDING node(IAD/7429c130-a6f9-4820-84d1-03e86418a847)
          Mar 10, 2017 9:08:37 PM org.jclouds.logging.jdk.JDKLogger logInfo
          INFO: << customized node(IAD/7429c130-a6f9-4820-84d1-03e86418a847)
          Mar 10, 2017 9:08:37 PM jenkins.plugins.jclouds.compute.JCloudsSlave createComputer
          INFO: Creating a new JClouds Slave
          Mar 10, 2017 9:08:37 PM jenkins.plugins.jclouds.compute.PhoneHomeMonitor waitForPhoneHome
          INFO: Waiting for stuff-ba2 to phone home. 900 seconds until timeout.

          As I was not able to ssh the node, I imaged it and spawn a new node based on the image but I didn't find a reference to my cloud-init setting on this debug image (I looked in /var/lib/cloud/instance/cloud-config.txt as well as /var/log/cloud-init.log). How can I know if I did my config correctly ?{{}}

          Show
          vaceletm Manuel Vacelet added a comment - Thanks for the follow-up It should all have been automatically migrated. Didn't it do that? Sort of, the elements related to rackspace APIs seems to have been properly migrated (as I can see server beeing created) but for the rest it's unclear. The admin credentials are used for provisioning resp. running the init-script and can be either SSH-credentials or Username/Password credentials (depends on Provider, I don't have a rackspace account so I don't know the specifics there) Maybe there is an issue there because I either tried to: Leave it blank (as it was before upgrade) Set root + private key And none of these worked. So I gave a try to cloud-init, it seems to work better (at least the node is not terminated because jenkins cannot ssh it) but I didn't manage to got it right. Here was I now have in the logs: Mar 10, 2017 9:08:11 PM org.jclouds.logging.jdk.JDKLogger logInfo INFO: >> creating new server region(IAD) name(stuff-ba2) image(9ab07706-4b9f-41ad-a8f5-8b4fda2e40f7) flavor(general1-1) options(CreateServerOptions{keyName=null, metadata={Name=stuff, jclouds-group=stuff}, userData=#cloud-config users: - name: jenkins {{ homedir: /jenkins}} {{ ssh-authorized-keys:}} {{ - ssh-rsa AAAAB....}} {{ sudo: ['ALL=(ALL) NOPASSWD:ALL'] }} {{ groups: sudo}} {{ shell: /bin/bash}} phone_home: {{ url: https://ci.tuleap.org/jenkins/jclouds-phonehome/ }} {{ tries: 3}} {{ , availabilityZone=null, configDrive=false})}} Mar 10, 2017 9:08:14 PM org.jclouds.logging.jdk.JDKLogger logInfo INFO: << PENDING node(IAD/7429c130-a6f9-4820-84d1-03e86418a847) Mar 10, 2017 9:08:37 PM org.jclouds.logging.jdk.JDKLogger logInfo INFO: << customized node(IAD/7429c130-a6f9-4820-84d1-03e86418a847) Mar 10, 2017 9:08:37 PM jenkins.plugins.jclouds.compute.JCloudsSlave createComputer INFO: Creating a new JClouds Slave Mar 10, 2017 9:08:37 PM jenkins.plugins.jclouds.compute.PhoneHomeMonitor waitForPhoneHome INFO: Waiting for stuff-ba2 to phone home. 900 seconds until timeout. As I was not able to ssh the node, I imaged it and spawn a new node based on the image but I didn't find a reference to my cloud-init setting on this debug image (I looked in  /var/lib/cloud/instance/cloud-config.txt as well as  /var/log/cloud-init.log). How can I know if I did my config correctly ? {{}}
          Hide
          felfert Fritz Elfert added a comment - - edited

          Did you literally used those double curly braces? If yes, that's completely wrong. Where did you get that from?
          Contact me via the email listed in my github-profile and I can send you some working examples.
          BTW: cloud-init is quite picky about indentation.

          Edit:
          Did the cloud-init log file exist at least? If not: Maybe your specific image does not have cloud-init incuded?

          Just seen: In that blog, they mention using a config-drive. You should probably ask them if this works with regular metadata as well.

          Show
          felfert Fritz Elfert added a comment - - edited Did you literally used those double curly braces? If yes, that's completely wrong. Where did you get that from? Contact me via the email listed in my github-profile and I can send you some working examples. BTW: cloud-init is quite picky about indentation. Edit: Did the cloud-init log file exist at least? If not: Maybe your specific image does not have cloud-init incuded? Just seen: In that blog, they mention using a config-drive. You should probably ask them if this works with regular metadata as well.
          Hide
          felfert Fritz Elfert added a comment -

          If both the admin username and the admin password were empty before, Set the admin credential to None
          If the admin username was set and the admin password was empty before, then the admin credential should be a SSH-credential with the same username and a copy of the global cloud-key as private key.
          If both admin user and admin password were set before, then the admin credential should be a Username/Password credential using this username and password.

          Show
          felfert Fritz Elfert added a comment - If both the admin username and the admin password were empty before, Set the admin credential to None If the admin username was set and the admin password was empty before, then the admin credential should be a SSH-credential with the same username and a copy of the global cloud-key as private key. If both admin user and admin password were set before, then the admin credential should be a Username/Password credential using this username and password.
          Hide
          felfert Fritz Elfert added a comment -

          Do you know of a way to get a free trial account at rackspace? If yes, then I could test all that stuff myself.

          Show
          felfert Fritz Elfert added a comment - Do you know of a way to get a free trial account at rackspace? If yes, then I could test all that stuff myself.
          Hide
          vaceletm Manuel Vacelet added a comment - - edited

          Did you literally used those double curly braces? If yes, that's completely wrong. Where did you get that from?

          Nop, I got a yaml (verified with yaml linter) and those curly braces are what I got from jenkins logs when pasting in jira.

          Did the cloud-init log file exist at least? If not: Maybe your specific image does not have cloud-init incuded?

          There is a cloud init log (ubuntu 14.04 image) but nothing mention the actions of my cloud init config.

          Show
          vaceletm Manuel Vacelet added a comment - - edited Did you literally used those double curly braces? If yes, that's completely wrong. Where did you get that from? Nop, I got a yaml (verified with yaml linter) and those curly braces are what I got from jenkins logs when pasting in jira. Did the cloud-init log file exist at least? If not: Maybe your specific image does not have cloud-init incuded? There is a cloud init log (ubuntu 14.04 image) but nothing mention the actions of my cloud init config.
          Hide
          vaceletm Manuel Vacelet added a comment -

          I managed to get cloud-init working, I had to enable "config drive" option in jcloud config

          Show
          vaceletm Manuel Vacelet added a comment - I managed to get cloud-init working, I had to enable "config drive" option in jcloud config
          Hide
          vaceletm Manuel Vacelet added a comment -

          So I confirm using cloudinit + config drive + "use pre-existing jenkins user" solves the issue.

          Show
          vaceletm Manuel Vacelet added a comment - So I confirm using cloudinit + config drive + "use pre-existing jenkins user" solves the issue.
          Hide
          felfert Fritz Elfert added a comment -

          Since the old style provisioning will be deprecated in the next release anyway, I'm closing this as resolved.

          Show
          felfert Fritz Elfert added a comment - Since the old style provisioning will be deprecated in the next release anyway, I'm closing this as resolved.
          felfert Fritz Elfert made changes -
          Field Original Value New Value
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Done [ 10000 ]
          felfert Fritz Elfert made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              felfert Fritz Elfert
              Reporter:
              vaceletm Manuel Vacelet
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: