Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-42676

Unable to ssh provisionned node on openstack (rackspace) after 2.8 -> 2.14 upgrade

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Component/s: jclouds-plugin
    • Labels:
      None
    • Environment:
      java version "1.7.0_121"
      OpenJDK Runtime Environment (IcedTea 2.6.8) (7u121-2.6.8-1ubuntu0.12.04.3)
      OpenJDK 64-Bit Server VM (build 24.121-b00, mixed mode)
      Ubuntu 12.04
      jclouds-plugin 2.14
      jenkins 2.32.3
    • Similar Issues:

      Description

      After upgrade from 2.8.1- to 2.14 I can no longer provision nodes on rackspace.

      First, I had to remove "Assign Public IP" (previously on) otherwise I got "java.lang.IllegalArgumentException: Floating IPs are required by options, but the extension is not available! options:"

      Then, even after having remove that option, I cannot find a combination of credentials that allows jenkins to ssh the server to run the init script.

      The connection to rackspace does work (tested from the interface) and the hardware & image are correct because I can see in my console that the server is being provisioned.

      But once it's up, jenkins cannot SSH:

      INFO: Provisioning new jclouds node
      Mar 10, 2017 5:22:02 PM jenkins.plugins.jclouds.compute.JCloudsSlaveTemplate get
      INFO: Setting osFamily to ubuntu
      Mar 10, 2017 5:22:02 PM jenkins.plugins.jclouds.compute.JCloudsSlaveTemplate get
      INFO: Setting osVersion to 14.04
      Mar 10, 2017 5:22:02 PM jenkins.plugins.jclouds.compute.JCloudsSlaveTemplate get
      INFO: Setting hardware Id to IAD/2
      Mar 10, 2017 5:22:02 PM org.jclouds.logging.jdk.JDKLogger logInfo
      INFO: >> searching params({osFamily=ubuntu, osVersion=14.04, hardwareId=IAD/2})
      Mar 10, 2017 5:22:02 PM org.jclouds.logging.jdk.JDKLogger logInfo
      INFO: <<   matched image(IAD/9ab07706-4b9f-41ad-a8f5-8b4fda2e40f7) hardware(IAD/2) location(IAD)
      Mar 10, 2017 5:22:02 PM jenkins.plugins.jclouds.compute.JCloudsSlaveTemplate get
      INFO: Setting adminCredentialsId to 502a5302-fb98-422a-a1dd-c8998d8c5e83
      Mar 10, 2017 5:22:02 PM jenkins.plugins.jclouds.compute.JCloudsSlaveTemplate get
      INFO: Using username/privatekey as adminCredentials
      Mar 10, 2017 5:22:02 PM org.jclouds.logging.jdk.JDKLogger logInfo
      INFO: >> running 1 node group(stuff) location(IAD) image(IAD/9ab07706-4b9f-41ad-a8f5-8b4fda2e40f7) hardwareProfile(IAD/2) options({loginUser=root, loginPrivateKeyPresent=true, scriptPresent=true, userMetadata={Name=stuff}, autoAssignFloati
      ngIp=false, configDrive=false})
      Mar 10, 2017 5:22:03 PM org.jclouds.logging.jdk.JDKLogger logInfo
      INFO: >> adding node location(IAD) name(stuff-1be) image(9ab07706-4b9f-41ad-a8f5-8b4fda2e40f7) hardware(2)
      Mar 10, 2017 5:22:03 PM org.jclouds.logging.jdk.JDKLogger logInfo
      INFO: >> creating new server region(IAD) name(stuff-1be) image(9ab07706-4b9f-41ad-a8f5-8b4fda2e40f7) flavor(2) options(CreateServerOptions{keyName=null, metadata={Name=stuff, jclouds-group=stuff}, userData=null, availabilityZone=null, conf
      igDrive=false})
      Mar 10, 2017 5:22:18 PM org.jclouds.logging.jdk.JDKLogger logInfo
      INFO: << PENDING node(IAD/a8ce5366-9635-40da-a250-ec0e8df597ff)
      Mar 10, 2017 5:30:15 PM com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.actions.manual.ManualTriggerAction getServerConfig
      SEVERE: Could not find server null
      Mar 10, 2017 5:31:20 PM org.jclouds.logging.jdk.JDKLogger logInfo
      INFO: >> blocking on sockets [104.130.231.245:22, 10.176.7.13:22] for 600000 MILLISECONDS
      Mar 10, 2017 5:31:34 PM org.jclouds.logging.jdk.JDKLogger logInfo
      INFO: << socket 10.176.7.13:22 opened
      Mar 10, 2017 5:31:34 PM net.schmizz.sshj.transport.TransportImpl init
      INFO: Client identity string: SSH-2.0-SSHJ_0_9_2
      Mar 10, 2017 5:31:34 PM net.schmizz.sshj.transport.TransportImpl init
      INFO: Server identity string: SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7
      Mar 10, 2017 5:31:34 PM net.schmizz.sshj.transport.TransportImpl$1 notifyDisconnect
      INFO: Disconnected - BY_APPLICATION
      Mar 10, 2017 5:31:34 PM org.jclouds.logging.jdk.JDKLogger logInfo
      INFO: << (root:rsa[fingerprint(e1:b3:f7:6b:c5:fd:83:6c:e9:46:a4:89:90:05:24:a0),sha1(e7:de:3c:34:a4:cb:7e:8b:50:95:4d:78:93:fa:9f:18:67:7d:b2:c1)]@10.176.7.13:22) error acquiring {hostAndPort=10.176.7.13:22, loginUser=root, ssh=null, conne
      ctTimeout=60000, sessionTimeout=60000} (attempt 1 of 7): Exhausted available authentication methods
      Mar 10, 2017 5:31:34 PM net.schmizz.sshj.transport.TransportImpl init
      INFO: Client identity string: SSH-2.0-SSHJ_0_9_2
      Mar 10, 2017 5:31:34 PM net.schmizz.sshj.transport.TransportImpl init
      INFO: Server identity string: SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7
      Mar 10, 2017 5:31:34 PM net.schmizz.sshj.transport.TransportImpl$1 notifyDisconnect
      INFO: Disconnected - BY_APPLICATION
      Mar 10, 2017 5:31:34 PM org.jclouds.logging.jdk.JDKLogger logInfo
      INFO: << (root:rsa[fingerprint(e1:b3:f7:6b:c5:fd:83:6c:e9:46:a4:89:90:05:24:a0),sha1(e7:de:3c:34:a4:cb:7e:8b:50:95:4d:78:93:fa:9f:18:67:7d:b2:c1)]@10.176.7.13:22) error acquiring {hostAndPort=10.176.7.13:22, loginUser=root, ssh=null, connectTimeout=60000, sessionTimeout=60000} (attempt 2 of 7): Exhausted available authentication methods
      Mar 10, 2017 5:31:38 PM net.schmizz.sshj.transport.TransportImpl init
      INFO: Client identity string: SSH-2.0-SSHJ_0_9_2
      Mar 10, 2017 5:31:38 PM net.schmizz.sshj.transport.TransportImpl init
      INFO: Server identity string: SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7
      Mar 10, 2017 5:31:38 PM net.schmizz.sshj.transport.TransportImpl$1 notifyDisconnect
      INFO: Disconnected - BY_APPLICATION

      That ends up to

      Mar 10, 2017 5:32:14 PM org.jclouds.logging.jdk.JDKLogger logError
      SEVERE: << (root:rsa[fingerprint(e1:b3:f7:6b:c5:fd:83:6c:e9:46:a4:89:90:05:24:a0),sha1(e7:de:3c:34:a4:cb:7e:8b:50:95:4d:78:93:fa:9f:18:67:7d:b2:c1)]@10.176.7.13:22) error acquiring {hostAndPort=10.176.7.13:22, loginUser=root, ssh=null, connectTimeout=60000, sessionTimeout=60000} (out of retries - max 7): Exhausted available authentication methods
      net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods

      The documentation is not clear on what is expected in term of credentials between

      • Cloud RSA (used to be private & public key, now we need to setup a username: which one ?)
      • Jenkins Credentials
      • Admin credentials

      Which one is used to run initscript and what should I set in term of usernames ?

        Attachments

          Activity

          Hide
          felfert Fritz Elfert added a comment -

          Do you know of a way to get a free trial account at rackspace? If yes, then I could test all that stuff myself.

          Show
          felfert Fritz Elfert added a comment - Do you know of a way to get a free trial account at rackspace? If yes, then I could test all that stuff myself.
          Hide
          vaceletm Manuel Vacelet added a comment - - edited

          Did you literally used those double curly braces? If yes, that's completely wrong. Where did you get that from?

          Nop, I got a yaml (verified with yaml linter) and those curly braces are what I got from jenkins logs when pasting in jira.

          Did the cloud-init log file exist at least? If not: Maybe your specific image does not have cloud-init incuded?

          There is a cloud init log (ubuntu 14.04 image) but nothing mention the actions of my cloud init config.

          Show
          vaceletm Manuel Vacelet added a comment - - edited Did you literally used those double curly braces? If yes, that's completely wrong. Where did you get that from? Nop, I got a yaml (verified with yaml linter) and those curly braces are what I got from jenkins logs when pasting in jira. Did the cloud-init log file exist at least? If not: Maybe your specific image does not have cloud-init incuded? There is a cloud init log (ubuntu 14.04 image) but nothing mention the actions of my cloud init config.
          Hide
          vaceletm Manuel Vacelet added a comment -

          I managed to get cloud-init working, I had to enable "config drive" option in jcloud config

          Show
          vaceletm Manuel Vacelet added a comment - I managed to get cloud-init working, I had to enable "config drive" option in jcloud config
          Hide
          vaceletm Manuel Vacelet added a comment -

          So I confirm using cloudinit + config drive + "use pre-existing jenkins user" solves the issue.

          Show
          vaceletm Manuel Vacelet added a comment - So I confirm using cloudinit + config drive + "use pre-existing jenkins user" solves the issue.
          Hide
          felfert Fritz Elfert added a comment -

          Since the old style provisioning will be deprecated in the next release anyway, I'm closing this as resolved.

          Show
          felfert Fritz Elfert added a comment - Since the old style provisioning will be deprecated in the next release anyway, I'm closing this as resolved.

            People

            • Assignee:
              felfert Fritz Elfert
              Reporter:
              vaceletm Manuel Vacelet
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: