Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-62724

EC2 plugin fails to get console output

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Not A Defect
    • Component/s: ec2-plugin
    • Labels:
    • Environment:
      Jenkins: 2.241
      EC2 Plugin 1.50.3
    • Similar Issues:

      Description

      I recently upgraded from 1.49.1. Now Jenkins cannot use any EC2 workers. I am using the latest standard AWS linux2 AMI.

      The node log says, over and over :

       

      INFO: Waiting for SSH to come up. Sleeping 5.
      Jun 19, 2020 10:54:07 AM hudson.plugins.ec2.EC2Cloud
      INFO: Connecting to 10.66.2.89 on port 22, with timeout 10000.
      Jun 19, 2020 10:54:07 AM hudson.plugins.ec2.EC2Cloud
      INFO: The instance EC2 (Jenkins) - Default Slave (i-04821a9ba3e3e5cf3) has a blank console. Maybe the console is yet not available. If enough time has passed, consider changing the key verification strategy or the AMI used by one printing out the host key in the instance console
      Jun 19, 2020 10:54:07 AM hudson.plugins.ec2.EC2Cloud
      INFO: The instance console is blank. Cannot check the key. The connection to EC2 (Jenkins) - Default Slave (i-04821a9ba3e3e5cf3) is not allowed
      Jun 19, 2020 10:54:07 AM hudson.plugins.ec2.EC2Cloud
      INFO: Failed to connect via ssh: There was a problem while connecting to 10.66.2.89:22
      Jun 19, 2020 10:54:07 AM hudson.plugins.ec2.EC2Cloud

       

      And yet, I can query the instance console log from the Jenkins master without any problems and see the ssh key sections (obfuscated here) :

      aws ec2 get-console-output --instance-id i-04821a9ba3e3e5cf3aws ec2 get-console-output --instance-id i-04821a9ba3e3e5cf3{    "InstanceId": "i-04821a9ba3e3e5cf3",     "Output": "                 102/117 \r\n  Installing : libcom_err-devel-1.43.5-2.43.amzn1.x86_64                103/117 \r\n  Installing : libverto-devel-0.2.5-4.9.amzn1.x86_64                    104/117 \r\n  Installing : libsepol-devel-2.1.7-3.12.amzn1.x86_64                   105/117 \r\n  Installing : libselinux-devel-2.1.10-3.22.amzn1.x86_64                106/117 \r\n  Installing : krb5-devel-1.15.1-46.48.amzn1.x86_64                     107/117 \r\n  Installing : 1:openssl-devel-1.0.2k-16.151.amzn1.x86_64               108/117 \r\n  Installing : nodejs-devel-0.10.48-3.el6.x86_64                        109/117 \r\n  Installing : node-gyp-0.10.6-2.el6.noarch                             110/117 \r\n  Installing : npm-1.3.6-5.el6.noarch                                   111/117 \r\n  Installing : gcc-c++-4.8.5-1.22.amzn1.noarch                          112/117 \r\n  Installing : jq-1.5-1.2.amzn1.x86_64                                  113/117 \r\n  Cleanup    : 1:openssl-1.0.2k-13.111.amzn1.x86_64                   ........
      {{ Jenkins2 slave \r\nec2: \r\nec2: #############################################################\r\nec2: ----BEGIN SSH HOST KEY FINGERPRINTS---\r\nec2: 1024 SHA256:+I3roaPC4ojmofR9/4a3oLGQF2N6gdbQ+HWP2J3ANKc root@ip-10-66-2-89 (DSA)\r\nec2: 256 SHA256:BCenkdcTlXQww4Rao/f+6+VxxxAp5OpB1g root@ip-10-66-2-89 (ECDSA)\r\nec2: 256 SHA256:YikvW/GI6ST7IH9YOxxxxu21nkVbvrb6SNSE no comment (ED25519)\r\nec2: 2048 SHA256:IhlV9fmdi3fWP/YOdlVmHtxxxxwNrQ6xuu9eM root@ip-10-66-2-89 (RSA)\r\nec2: -----END SSH HOST KEY FINGERPRINTS---\r\nec2: #############################################################\r\n---BEGIN SSH HOST KEY KEYS---\r\necdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHxxA6dZ0uFCxer5B+LL4BbucJXd7Us2Zet/jHxHdrTrSR9i1n475IsnNDk0+HKMIdqnSMpM8Q5W+yMnY= root@ip-10-66-2-89\r\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBnyi6P5k5EJSMROnadSBRclaqcA6cvuPJVJGLEDJ+xF \r\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEsJ3k9xzI1WkaxxxAms7UXStJjfDQus2xxcDe4DWl6ruYVjYxFXMJT3yLdhWyyGj02+WQjb61eFCoECcEMMEM+38ovYts3zkF8lXsc3eMVazmuAjJQgFQhmqWdWkn2iR/0Vbodb3ZMjWJZMYw9dTe0QFxxxXx9Lzi0RS0Yy2bxJITdjskj+aLDLIPwLHjNidHtBZKnB9H58t06aqdRusKw4lJKl5u7TgMQy4Ywrof2Dx46G8/RsvatVVesGtZ+JaB6AmXfLj/OW9OQ3aK/Ls/WAZpFTvslaxxXtXoYL4qh root@ip-10-66-2-89\r\n---END SSH HOST KEY KEYS----\r\nCloud-init v. 0.7.6 finished at Fri, 19 Jun 2020 10:54:34 +0000. Datasource DataSourceEc2.  Up 146.18 seconds\r\n\r\r\nAmazon Linux AMI release 2018.03\r\nKernel 4.14.77-70.59.amzn1.x86_64 on an x86_64\r\n\r\n",     "Timestamp": "2020-06-19T10:58:20.000Z"}}}

       

       

      I have changed  "Host Key Verification Strategy" from "Check-new-soft" to "off" and it works again. But if it thinks the console log is empty I cannot use any of the safer options for validating the key. (And have to change all my node definitions...)

      Is the plugin correctly using the instance role to get credentials to query the log? (guess?!)

        Attachments

          Activity

          Hide
          max_allan max allan added a comment -

          My bad. Or AWS's.
          The console log takes ages to populate. Over 5 minutes. Nearer 10. So I either have to change the launch time out or the security mode. (or maybe send a big chunk of output to the syslog to encourage it to flush the buffer more promptly)

          Show
          max_allan max allan added a comment - My bad. Or AWS's. The console log takes ages to populate. Over 5 minutes. Nearer 10. So I either have to change the launch time out or the security mode. (or maybe send a big chunk of output to the syslog to encourage it to flush the buffer more promptly)
          Hide
          mramonleon Ramon Leon added a comment -

          Sadly it's true. This fact is documented: https://github.com/jenkinsci/ec2-plugin/#check-new-hard

          The launch timeout should be long enough to allow the plugin to check the instance console. With this strategy, the plugin waits for the console to be available, which can take a few minutes. The Launch Timeout in seconds field should have a number to allow that, for example 600 (10 minutes). By default there is no timeout, so it's safe.

          That said, this waiting period only happens once, to retrieve the host key. Once the plugin gets the key, stores it to accept future connections to the instance. You can use Accept New which accepts blindly the first one Jenkins connects to the instance. It's not so safe, but at least prevents future MitM attacks.

          Show
          mramonleon Ramon Leon added a comment - Sadly it's true. This fact is documented: https://github.com/jenkinsci/ec2-plugin/#check-new-hard The launch timeout should be long enough to allow the plugin to check the instance console. With this strategy, the plugin waits for the console to be available, which can take a few minutes. The Launch Timeout in seconds field should have a number to allow that, for example 600 (10 minutes). By default there is no timeout, so it's safe. That said, this waiting period only happens once, to retrieve the host key. Once the plugin gets the key, stores it to accept future connections to the instance. You can use Accept New which accepts blindly the first one Jenkins connects to the instance. It's not so safe, but at least prevents future MitM attacks.

            People

            • Assignee:
              thoulen FABRIZIO MANFREDI
              Reporter:
              max_allan max allan
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: