Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43150

Jenkins vsphere cloud pluging connecting ssh to wrong ip when slave has multiple network interfaces

    Details

    • Similar Issues:

      Description

      Pluging getting the first ip reported by vsphere when host has multiple network interfaces, i attached a screenshot showing the different ips of the newly created host, when master try to ssh to the new hos to connect using ssh it use the first ip that vmware reports 172.18.0.1, instead of the recently assigined ip 10.7.119.11 corresponding to eth0, the pluging should have the ability to especify the ip from what interface you want to use to connect using ssh.

      below you will find the result of ifconfig from that host.

      also all those interfaces are created by docker and vmware.
       

      br-2726c7674fcb: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
              inet 172.18.0.1  netmask 255.255.0.0  broadcast 0.0.0.0
              ether 02:42:7c:98:e3:78  txqueuelen 0  (Ethernet)
              RX packets 0  bytes 0 (0.0 B)
              RX errors 0  dropped 0  overruns 0  frame 0
              TX packets 0  bytes 0 (0.0 B)
              TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
      
      br-8ded986f879b: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
              inet 11.11.1.1  netmask 255.255.255.0  broadcast 0.0.0.0
              ether 02:42:36:f4:3a:58  txqueuelen 0  (Ethernet)
              RX packets 57571  bytes 10953487 (10.4 MiB)
              RX errors 0  dropped 0  overruns 0  frame 0
              TX packets 92953  bytes 7961174 (7.5 MiB)
              TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
      
      br-a990d4d3236a: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
              inet 172.19.0.1  netmask 255.255.0.0  broadcast 0.0.0.0
              ether 02:42:67:9b:e3:db  txqueuelen 0  (Ethernet)
              RX packets 20344  bytes 2367279 (2.2 MiB)
              RX errors 0  dropped 0  overruns 0  frame 0
              TX packets 25976  bytes 8415939 (8.0 MiB)
              TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
      
      br-f846e442c25d: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
              inet 172.20.0.1  netmask 255.255.0.0  broadcast 0.0.0.0
              ether 02:42:1e:31:d3:7c  txqueuelen 0  (Ethernet)
              RX packets 223228  bytes 14865454 (14.1 MiB)
              RX errors 0  dropped 0  overruns 0  frame 0
              TX packets 6870  bytes 584216 (570.5 KiB)
              TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
      
      docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
              inet 172.17.0.1  netmask 255.255.0.0  broadcast 0.0.0.0
              ether 02:42:09:32:3a:ed  txqueuelen 0  (Ethernet)
              RX packets 0  bytes 0 (0.0 B)
              RX errors 0  dropped 0  overruns 0  frame 0
              TX packets 0  bytes 0 (0.0 B)
              TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
      
      eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
              inet 10.7.119.11  netmask 255.255.248.0  broadcast 10.7.119.255
              ether 00:50:56:99:c4:01  txqueuelen 1000  (Ethernet)
              RX packets 223228  bytes 14865454 (14.1 MiB)
              RX errors 0  dropped 0  overruns 0  frame 0
              TX packets 6870  bytes 584216 (570.5 KiB)
              TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
      
      lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
              inet 127.0.0.1  netmask 255.0.0.0
              loop  txqueuelen 1  (Local Loopback)
              RX packets 2298  bytes 92042 (89.8 KiB)
              RX errors 0  dropped 0  overruns 0  frame 0
              TX packets 2298  bytes 92042 (89.8 KiB)
              TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
      
      veth0ef36a1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
              ether 46:45:ad:37:66:78  txqueuelen 0  (Ethernet)
              RX packets 8  bytes 648 (648.0 B)
              RX errors 0  dropped 0  overruns 0  frame 0
              TX packets 0  bytes 0 (0.0 B)
              TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
      
      veth19e3aa1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
              ether 4a:43:ab:96:99:c7  txqueuelen 0  (Ethernet)
              RX packets 20344  bytes 2367279 (2.2 MiB)
              RX errors 0  dropped 0  overruns 0  frame 0
              TX packets 25976  bytes 8415939 (8.0 MiB)
              TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
      
      veth984e106: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
              ether 66:d1:5c:c2:fc:d5  txqueuelen 0  (Ethernet)
              RX packets 25976  bytes 8415939 (8.0 MiB)
              RX errors 0  dropped 0  overruns 0  frame 0
              TX packets 20343  bytes 2367189 (2.2 MiB)
              TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
      
      vetha6d0f87: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
              ether 52:b8:89:99:74:f5  txqueuelen 0  (Ethernet)
              RX packets 57571  bytes 10953487 (10.4 MiB)
              RX errors 0  dropped 0  overruns 0  frame 0
              TX packets 92953  bytes 7961174 (7.5 MiB)
              TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
      

        Attachments

          Activity

          Hide
          pjdarton pjdarton added a comment -

          Hmm, this is going to be tricky - the vSphere plugin has no idea which IP address is going to be the "correct" one to use (which is why it assumes that the "first" will be acceptable).

          What I would suggest is that you use JNLP instead - make the slave connect to the master.  That way it's up to the slave's network layer to work out how to talk to Jenkins instead of asking the vSphere plugin code to second-guess which option might be correct.  This would also give you the bonus that it'll work even if the slave is on a Host-local network which only gives NAT access to the wider network.

          Show
          pjdarton pjdarton added a comment - Hmm, this is going to be tricky - the vSphere plugin has no idea which IP address is going to be the "correct" one to use (which is why it assumes that the "first" will be acceptable). What I would suggest is that you use JNLP instead - make the slave connect to the master.  That way it's up to the slave's network layer to work out how to talk to Jenkins instead of asking the vSphere plugin code to second-guess which option might be correct.  This would also give you the bonus that it'll work even if the slave is on a Host-local network which only gives NAT access to the wider network.
          Hide
          rpocase Robby Pocase added a comment -

          pjdarton Is there a concrete set of steps one should take for configuring JNLP? The Guest Info portion has some vague recommendations, but isn't concrete on how a node would take advantaged of the injected info (i.e., should I add an /etc/rc* file to assume that information is available)?

          Show
          rpocase Robby Pocase added a comment - pjdarton Is there a concrete set of steps one should take for configuring JNLP? The Guest Info portion has some vague recommendations, but isn't concrete on how a node would take advantaged of the injected info (i.e., should I add an /etc/rc* file to assume that information is available)?
          Hide
          pjdarton pjdarton added a comment -

          Yes; you create a script that will interrogate vSphere to obtain the GuestInfo information and then launch the slave, and you ensure that the guest OS runs that script automatically when it boots up.
          Exactly how you do that will vary from guest OS to guest OS, e.g. the techniques used on Microsoft Windows are very different to those used on Ubuntu Linux, but the end result remains the same.

          Show
          pjdarton pjdarton added a comment - Yes; you create a script that will interrogate vSphere to obtain the GuestInfo information and then launch the slave, and you ensure that the guest OS runs that script automatically when it boots up. Exactly how you do that will vary from guest OS to guest OS, e.g. the techniques used on Microsoft Windows are very different to those used on Ubuntu Linux, but the end result remains the same.
          Hide
          rpocase Robby Pocase added a comment - - edited

          pjdarton That's what I ended up coming up with yesterday afternoon. My solution is based around Ubuntu 16.04, but posting as others might find it useful. The error checking is pretty minimal but should be sufficient for most. I injected JNLPURL, JENKINS_URL, and JNLP_SECRET per the help text.

           

          #!/bin/bash -e
          #
          # /etc/rc.local
          #
          # This script is executed at the end of each multiuser runlevel (e.g. on first boot). 
          # Make sure that the script will "exit 0" on success or any other
          # value on error.
          # 
          # On ubuntu 16.04, must enable rc.local service (systemctl enable rc.local)
          
          JENKINS_URL=$(vmtoolsd --cmd "info-get guestinfo.JENKINS_URL")                                                                                                                                                      
          case "$JENKINS_URL" in                                                                                                                                                                                              
            http*)                                                                                                                                                                                                            
              JNLPURL=$(vmtoolsd --cmd "info-get guestinfo.JNLPURL")                                                                                                                                                          
              JNLP_SECRET=$(vmtoolsd --cmd "info-get guestinfo.JNLP_SECRET")
              #JAVA_OPTS MUST be set to avoid vmtoolsd error                                                                                                                                                  
              JAVA_OPTS=$(vmtoolsd --cmd "info-get guestinfo.JAVA_OPTS")                                                                                                                                                      
              wget "${JENKINS_URL}/jnlpJars/slave.jar" -O /tmp/slave.jar                                                                                                                                                      
              java $JAVA_OPTS -jar /tmp/slave.jar -jnlpUrl $JNLPURL -secret $JNLP_SECRET                                                                                                                                      
              ;;                                                                                                                                                                                                              
            *)                                                                                                                                                                                                                
              echo "Does not appear to be running in Jenkins"                                                                                                                                                                 
              exit 1                                                                                                                                                                                                          
              ;;                                                                                                                                                                                                              
          esac
          
          exit 0
          
          
          Show
          rpocase Robby Pocase added a comment - - edited pjdarton That's what I ended up coming up with yesterday afternoon. My solution is based around Ubuntu 16.04, but posting as others might find it useful. The error checking is pretty minimal but should be sufficient for most. I injected JNLPURL, JENKINS_URL, and JNLP_SECRET per the help text.   #!/bin/bash -e # # /etc/rc.local # # This script is executed at the end of each multiuser runlevel (e.g. on first boot). # Make sure that the script will "exit 0" on success or any other # value on error. # # On ubuntu 16.04, must enable rc.local service (systemctl enable rc.local) JENKINS_URL=$(vmtoolsd --cmd "info-get guestinfo.JENKINS_URL" ) case "$JENKINS_URL" in http*) JNLPURL=$(vmtoolsd --cmd "info-get guestinfo.JNLPURL" ) JNLP_SECRET=$(vmtoolsd --cmd "info-get guestinfo.JNLP_SECRET" ) #JAVA_OPTS MUST be set to avoid vmtoolsd error JAVA_OPTS=$(vmtoolsd --cmd "info-get guestinfo.JAVA_OPTS" ) wget "${JENKINS_URL}/jnlpJars/slave.jar" -O /tmp/slave.jar java $JAVA_OPTS -jar /tmp/slave.jar -jnlpUrl $JNLPURL -secret $JNLP_SECRET ;; *) echo "Does not appear to be running in Jenkins" exit 1 ;; esac exit 0
          Hide
          pjdarton pjdarton added a comment -

          Personally I would also have an extra parameter to allow you to inject additional JVM arguments e.g. -Xms... and -Dproperty=value etc.
          You might not need them right now, but there will come a time, and it's much easier to edit the values provided by the GuestInfo configuration than it is to rebuild your slave VM image.

          Also (and this is a personal code-style preference) I'd suggest you wrap the ${JNLPURL} and ${JNLP_SECRET} arguments in "quotes" - you never know when someone might put a space in some bit of data.

          Show
          pjdarton pjdarton added a comment - Personally I would also have an extra parameter to allow you to inject additional JVM arguments e.g. -Xms... and -Dproperty=value etc. You might not need them right now, but there will come a time, and it's much easier to edit the values provided by the GuestInfo configuration than it is to rebuild your slave VM image. Also (and this is a personal code-style preference) I'd suggest you wrap the ${JNLPURL} and ${JNLP_SECRET} arguments in "quotes" - you never know when someone might put a space in some bit of data.
          Hide
          rpocase Robby Pocase added a comment -

          pjdarton Thanks for the tip! Huge oversight on my end due to being so used to typical slave configuration. Will definitely get that added to the VM definition as an optional value (and update the config in line when I do so). I definitely agree on the style recommendations as well.

          Show
          rpocase Robby Pocase added a comment - pjdarton Thanks for the tip! Huge oversight on my end due to being so used to typical slave configuration. Will definitely get that added to the VM definition as an optional value (and update the config in line when I do so). I definitely agree on the style recommendations as well.

            People

            • Assignee:
              Unassigned
              Reporter:
              joaquingt joaquin mendez
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: