Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-62195

ec2-1.50.2 doesn't work with SSH <7.5

    Details

    • Similar Issues:
    • Released As:
      EC2 Plugin 1.50.3

      Description

      Version 1.50.2 introduces security mitigations by proposing new options for SSH.

      2 of the 3 options have been introduced by SSH version 7.6:

      • ssh(1): expand the StrictHostKeyChecking option with two new
        settings. The first "accept-new" will automatically accept
        hitherto-unseen keys but will refuse connections for changed or
        invalid hostkeys. This is a safer subset of the current behaviour
        of StrictHostKeyChecking=no. The second setting "off", is a synonym
        for the current behaviour of StrictHostKeyChecking=no: accept new
        host keys, and continue connection for hosts with incorrect
        hostkeys. A future release will change the meaning of
        StrictHostKeyChecking=no to the behaviour of "accept-new". bz#2400

      Although it was released almost 3 years ago, this seriously breaks compatibility with non-recent Jenkins installations.

      For instance, the current default Docker image for Jenkins is currently based off Debian Stretch which provides SSH 7.4 and doesn't support these new options:

      $ docker run --rm -ti jenkins/jenkins:2.235 ssh -o StrictHostKeyChecking=off
      command-line line 0: unsupported option "off".
      $ docker run --rm -ti jenkins/jenkins:2.235 ssh -o StrictHostKeyChecking=accept-new
      command-line line 0: unsupported option "accept-new".
      $ docker run --rm -ti jenkins/jenkins:lts ssh -o StrictHostKeyChecking=accept-new
      command-line line 0: unsupported option "accept-new".
      

        Attachments

          Issue Links

            Activity

            Hide
            oleg_nenashev Oleg Nenashev added a comment -
            Show
            oleg_nenashev Oleg Nenashev added a comment - https://github.com/jenkinsci/ec2-plugin/commit/91f48a7eb7aa1270970b92ece38606a97543deae  includes it. Several pull requests were missing in the release draft I fixed the changelog:  https://github.com/jenkinsci/ec2-plugin/releases/tag/ec2-1.50.3
            Hide
            multani Jonathan Ballet added a comment -

            Thanks Oleg Nenashev for the updated changelog!

            Show
            multani Jonathan Ballet added a comment - Thanks Oleg Nenashev for the updated changelog!
            Hide
            kanshi Antoine Hamon added a comment -

             
            I was able to reproduce this issue with Jenkins 2.237 (official docker image) & EC2 plugin 1.50.3
            After upgrading the server (from version 2.233), slaves appeared offlines and Jenkins were not able to re-connect to them:

            May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud
            INFO: Launching instance: i-xxxxxxx
            May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud
            INFO: bootstrap()
            May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud
            INFO: Getting keypair...
            May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud
            INFO: Using private key jenkins_slaves_new (SHA-1 fingerprint xx:xx:xx:xx:xx:xx:xx:xx)
            May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud
            INFO: Authenticating as centos
            May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud
            INFO: Connecting to 10.100.11.222 on port 22, with timeout 10000.
            May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud
            INFO: Connection allowed after the host key has been verified
            May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud
            INFO: Connected via SSH.
            May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud
            INFO: connect fresh as root
            May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud
            INFO: Connecting to 10.100.11.222 on port 22, with timeout 10000.
            May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud
            INFO: Connection allowed after the host key has been verified
            May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud
            INFO: Connected via SSH.
            May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud
            INFO: Creating tmp directory (/tmp) if it does not existMay 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud
            INFO: Verifying: java -fullversion
            openjdk full version "1.8.0_252-b09"
            May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud
            INFO: Verifying: which scp
            /usr/bin/scp
            May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud
            INFO: Copying remoting.jar to: /tmp
            May 22, 2020 5:43:34 PM hudson.plugins.ec2.EC2Cloud
            INFO: Launching remoting agent (via SSH client process): ssh -o StrictHostKeyChecking=accept-new -i /tmp/ec2_1385185844708247542.pem centos@10.100.11.222 -p 22  java  -jar /tmp/remoting.jar -workDir /jenkins
            [05/22/20 17:43:34] Launching agent
            $ ssh -o StrictHostKeyChecking=accept-new -i /tmp/ec2_1385185844708728885.pem centos@10.100.11.222 -p 22  java  -jar /tmp/remoting.jar -workDir /jenkins
            command-line line 0: unsupported option "accept-new".
            ERROR: Unable to launch the agent for EC2 (Slaves) - Slave (i-xxxxxxx)
            java.io.EOFException: unexpected stream termination
            	at hudson.remoting.ChannelBuilder.negotiate(ChannelBuilder.java:415)
            	at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:360)
            	at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:423)
            	at hudson.slaves.CommandLauncher.launch(CommandLauncher.java:165)
            	at hudson.plugins.ec2.ssh.EC2UnixLauncher.launchScript(EC2UnixLauncher.java:257)
            	at hudson.plugins.ec2.EC2ComputerLauncher.launch(EC2ComputerLauncher.java:48)
            	at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:296)
            	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
            	at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
            	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            	at java.lang.Thread.run(Thread.java:748)
            

            We still see the ssh -o StrictHostKeyChecking=accept-new

            Show
            kanshi Antoine Hamon added a comment -   I was able to reproduce this issue with Jenkins 2.237 (official docker image) & EC2 plugin 1.50.3 After upgrading the server (from version 2.233), slaves appeared offlines and Jenkins were not able to re-connect to them: May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud INFO: Launching instance: i-xxxxxxx May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud INFO: bootstrap() May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud INFO: Getting keypair... May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud INFO: Using private key jenkins_slaves_new (SHA-1 fingerprint xx:xx:xx:xx:xx:xx:xx:xx) May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud INFO: Authenticating as centos May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud INFO: Connecting to 10.100.11.222 on port 22, with timeout 10000. May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud INFO: Connection allowed after the host key has been verified May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud INFO: Connected via SSH. May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud INFO: connect fresh as root May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud INFO: Connecting to 10.100.11.222 on port 22, with timeout 10000. May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud INFO: Connection allowed after the host key has been verified May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud INFO: Connected via SSH. May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud INFO: Creating tmp directory (/tmp) if it does not existMay 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud INFO: Verifying: java -fullversion openjdk full version "1.8.0_252-b09" May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud INFO: Verifying: which scp /usr/bin/scp May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud INFO: Copying remoting.jar to: /tmp May 22, 2020 5:43:34 PM hudson.plugins.ec2.EC2Cloud INFO: Launching remoting agent (via SSH client process): ssh -o StrictHostKeyChecking=accept-new -i /tmp/ec2_1385185844708247542.pem centos@10.100.11.222 -p 22 java -jar /tmp/remoting.jar -workDir /jenkins [05/22/20 17:43:34] Launching agent $ ssh -o StrictHostKeyChecking=accept-new -i /tmp/ec2_1385185844708728885.pem centos@10.100.11.222 -p 22 java -jar /tmp/remoting.jar -workDir /jenkins command-line line 0: unsupported option "accept-new". ERROR: Unable to launch the agent for EC2 (Slaves) - Slave (i-xxxxxxx) java.io.EOFException: unexpected stream termination at hudson.remoting.ChannelBuilder.negotiate(ChannelBuilder.java:415) at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:360) at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:423) at hudson.slaves.CommandLauncher.launch(CommandLauncher.java:165) at hudson.plugins.ec2.ssh.EC2UnixLauncher.launchScript(EC2UnixLauncher.java:257) at hudson.plugins.ec2.EC2ComputerLauncher.launch(EC2ComputerLauncher.java:48) at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:296) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) We still see the  ssh -o StrictHostKeyChecking=accept-new
            Hide
            multani Jonathan Ballet added a comment -

            Antoine Hamon If you are using that old version of SSH (which doesn't support these options), you sadly need to keep the Host Key Verification Strategy setting of your AMI templates to off (the least secure, but the only compatible version).
            AFAIK, this was the value used before the new release of this plugin anyway.

            Show
            multani Jonathan Ballet added a comment - Antoine Hamon If you are using that old version of SSH (which doesn't support these options), you sadly need to keep the Host Key Verification Strategy setting of your AMI templates to off (the least secure, but the only compatible version). AFAIK, this was the value used before the new release of this plugin anyway.
            Hide
            kanshi Antoine Hamon added a comment -

            Thanks Jonathan Ballet I indeed missed that

            Show
            kanshi Antoine Hamon added a comment - Thanks Jonathan Ballet I indeed missed that

              People

              • Assignee:
                mramonleon Ramon Leon
                Reporter:
                multani Jonathan Ballet
              • Votes:
                4 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: