Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-42959

Failed known_hosts verification for SSH agent

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: ssh-slaves-plugin
    • Labels:
    • Environment:
      Versions:

      Jenkins: 2.51
      SSH Agent Plugin: 1.14
      SSH Credentials plugin: 1.13
      SSH Slaves Plugin : 1.15

      Ubuntu 14.04, 16.04
    • Similar Issues:

      Description

      SSH agent isn't launched after the latest update, complaining about missing records in the known hosts file.

      Nevertheless, the records do exist. I've tried to connect manually with ssh, everything was fine, ssh reports that it has found the host in known_hosts.

      Probable reason is the hashed host name in known_hosts.

      Here is the log from the Jenkins slave launch page:

      [03/21/17 11:40:34] [SSH] Opening SSH connection to xxx.xxx.xxx.xxx:22
      [03/21/17 11:40:34] [SSH] WARNING: No entry currently exists in the Known Hosts file for this host.      Connections will be denied until this new host and its associated key is added to the Known Hosts file.
           Key exchange was not finished, connection is closed.
           java.io.IOException: There was a problem while connecting to xxx.xxx.xxx.xxx:22
      	at com.trilead.ssh2.Connection.connect(Connection.java:818)
      	at com.trilead.ssh2.Connection.connect(Connection.java:687)
      	at com.trilead.ssh2.Connection.connect(Connection.java:601)
      	at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:1265)
      	at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:790)
      	at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:785)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
           Caused by: java.io.IOException: Key exchange was not finished, connection is closed.
      	at com.trilead.ssh2.transport.KexManager.getOrWaitForConnectionInfo(KexManager.java:93)
      	at com.trilead.ssh2.transport.TransportManager.getConnectionInfo(TransportManager.java:230)
      	at com.trilead.ssh2.Connection.connect(Connection.java:770)
      	... 9 more
           Caused by: java.io.IOException: The server hostkey was not accepted by the verifier callback
      	at com.trilead.ssh2.transport.KexManager.handleMessage(KexManager.java:591)
      	at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:777)
      	at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:489)
      	... 1 more
           [03/21/17 11:40:34] Launch failed - cleaning up connection
           [03/21/17 11:40:34] [SSH] Connection closed.
      
      

      And this is the debug output from ssh, showing that it has found a record in the second line of known_hosts.

        $ sudo -u jenkins -g jenkins ssh -v jenkins@xxx.xxx.xxx.xxx
          OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
          debug1: Reading configuration data /etc/ssh/ssh_config
          debug1: /etc/ssh/ssh_config line 19: Applying options for *
          debug1: Connecting to xxx.xxx.xxx.xxx [xxx.xxx.xxx.xxx] port 22.
          debug1: Connection established.
          debug1: identity file /var/lib/jenkins/.ssh/id_rsa type 1
      ...
          debug1: sending SSH2_MSG_KEX_ECDH_INIT
          debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
          debug1: Server host key: ECDSA xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx
          debug1: Host 'xxx.xxx.xxx.xxx.xxx' is known and matches the ECDSA host key.
          debug1: Found key in /var/lib/jenkins/.ssh/known_hosts:2
          debug1: ssh_ecdsa_verify: signature correct
      ...
          Welcome to Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-67-generic x86_64)
      

      Additional logs from Jenkins system log:

      Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
          kex_algo=diffie-hellman-group14-sha1
      Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
          server_host_key_algo=ssh-rsa
          Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
          enc_algo_client_to_server=aes256-ctr
          Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
          enc_algo_server_to_client=aes256-ctr
          Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
          mac_algo_client_to_server=hmac-sha1
          Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
          mac_algo_server_to_client=hmac-sha1
          Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
          comp_algo_client_to_server=none
          Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
          comp_algo_server_to_client=none
          Mar 21, 2017 12:06:08 PM FINE com.trilead.ssh2.transport.TransportManager
          Receive thread: error in receiveLoop
          java.io.IOException: The server hostkey was not accepted by the verifier callback
                  at com.trilead.ssh2.transport.KexManager.handleMessage(KexManager.java:591)
                  at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:777)
                  at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:489)
                  at java.lang.Thread.run(Thread.java:745)
      
          Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.TransportManager
          Receive thread: back from receiveLoop
          Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
          kex_algo=diffie-hellman-group14-sha1
          Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
          server_host_key_algo=ssh-rsa
          Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
          enc_algo_client_to_server=aes256-ctr
          Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
          enc_algo_server_to_client=aes256-ctr
          Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
          mac_algo_client_to_server=hmac-sha1
          Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
          mac_algo_server_to_client=hmac-sha1
          Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
          comp_algo_client_to_server=none
          Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
          comp_algo_server_to_client=none
          Mar 21, 2017 12:06:10 PM FINE com.trilead.ssh2.transport.TransportManager
          Receive thread: error in receiveLoop
          java.io.IOException: The server hostkey was not accepted by the verifier callback
                  at com.trilead.ssh2.transport.KexManager.handleMessage(KexManager.java:591)
                  at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:777)
                  at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:489)
                  at java.lang.Thread.run(Thread.java:745)
      
          Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.TransportManager
          Receive thread: back from receiveLoop
      

      File /var/lib/jenkins/.ssh/known_hosts contains strings, looking like being base64-encoded, delimited by '|'. Here is the sample.

      |1|DAg  ...   o... 1ll9wI=| ...  ....  tIrM= ecdsa-sha2-nistp256 xxxxxx..... bmlzdHAyNTYAAAAIbm................. .................xxxxxxxxxxxoKEHF3Vr0q685jI2+6vWjvAAG4lz5Ckujy9k=
      

      Github issue

        Attachments

          Issue Links

            Activity

            Hide
            danielbeck Daniel Beck added a comment -

            Olena Gaydarenko Please provide more information, such as the exact error message (with stack trace), as well as what the known_hosts entry looks like.

            Show
            danielbeck Daniel Beck added a comment - Olena Gaydarenko Please provide more information, such as the exact error message (with stack trace), as well as what the known_hosts entry looks like.
            Hide
            mcsf M Chon added a comment -

            I have these versions:

            Jenkins 2.60.2

            SSH Slaves Plugin 1.20

            and am still getting this error:

             

            Warning: no key algorithms provided; JENKINS-42959 disabled [08/14/17 18:47:10] [SSH] Opening SSH connection to xxxxxx:22. [08/14/17 18:47:10] [SSH] WARNING: No entry currently exists in the Known Hosts file for this host. Connections will be denied until this new host and its associated key is added to the Known Hosts file.

            Show
            mcsf M Chon added a comment - I have these versions: Jenkins 2.60.2 SSH Slaves Plugin 1.20 and am still getting this error:   Warning: no key algorithms provided; JENKINS-42959 disabled [08/14/17 18:47:10] [SSH] Opening SSH connection to xxxxxx:22. [08/14/17 18:47:10] [SSH] WARNING: No entry currently exists in the Known Hosts file for this host. Connections will be denied until this new host and its associated key is added to the Known Hosts file.
            Hide
            mwebber Matthew Webber added a comment -

            This message means exactly what it says.

            No entry currently exists in the Known Hosts file for this host. Connections will be denied until this new host and its associated key is added to the Known Hosts file

            In your agent configuration, for "Host Key Verification Strategy" you probably have "Known hosts file verification strategy". The help text for that says:

            Checks the known_hosts file (~/.ssh/known_hosts) for the user Jenkins is executing under, to see if an entry exists that matches the current connection.

            This method does not make any updates to the Known Hosts file, instead using the file as a read-only source and expecting someone with suitable access to the appropriate user account on the Jenkins master to update the file as required, potentially using the ssh hostname command to initiate a connection and update the file appropriately.

             So that's what you need to do. If you need more help, ask on the mailing list, not here, since it's not a bug.

            Show
            mwebber Matthew Webber added a comment - This message means exactly what it says. No entry currently exists in the Known Hosts file for this host. Connections will be denied until this new host and its associated key is added to the Known Hosts file In your agent configuration, for "Host Key Verification Strategy" you probably have "Known hosts file verification strategy". The help text for that says: Checks the known_hosts file (~/.ssh/known_hosts) for the user Jenkins is executing under, to see if an entry exists that matches the current connection. This method does not make any updates to the Known Hosts file, instead using the file as a read-only source and expecting someone with suitable access to the appropriate user account on the Jenkins master to update the file as required, potentially using the ssh hostname command to initiate a connection and update the file appropriately.  So that's what you need to do. If you need more help, ask on the mailing list, not here, since it's not a bug.
            Hide
            mcsf M Chon added a comment - - edited

            Yes, and I've been asking our elusive Sys Admin to please ssh from one host to the other in order to populate the known_hosts. This never used to be required by the plugin in the past (or if it was, I was not aware), but I guess it is now.

             

            Meanwhile, I noticed there is a dropdown menu under Node configuration, and I selected "Manually provided key Verification Strategy", and then I pasted the contents of id_rsa.pub into the "SSH Key" textbox, but got this error:

             

            [08/17/17 10:48:58] [SSH] WARNING: The SSH key for this host does not match the key required in the connection configuration. Connections will be denied until until the host key matches the configuration key. Key exchange was not finished, connection is closed.

            I then tried "Non verifying Verification Strategy", and got this error:

            [08/17/17 10:46:28] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection. ERROR: Server rejected the 1 private key(s) for jenkins (credentialId:e85865a0-e6eb-4de3-a77e-b0fbbc9ec8c4/method:publickey) [08/17/17 10:46:28] [SSH] Authentication failed. Authentication failed.

             

            So it seems like the "Manually provided key" method will work for me, but there is an issue with key mismatch. So, I'll follow up with our sys admin.

            Show
            mcsf M Chon added a comment - - edited Yes, and I've been asking our elusive Sys Admin to please ssh from one host to the other in order to populate the known_hosts. This never used to be required by the plugin in the past (or if it was, I was not aware), but I guess it is now.   Meanwhile, I noticed there is a dropdown menu under Node configuration, and I selected "Manually provided key Verification Strategy", and then I pasted the contents of id_rsa.pub into the "SSH Key" textbox, but got this error:   [08/17/17 10:48:58] [SSH] WARNING: The SSH key for this host does not match the key required in the connection configuration. Connections will be denied until until the host key matches the configuration key. Key exchange was not finished, connection is closed. I then tried "Non verifying Verification Strategy", and got this error: [08/17/17 10:46:28] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection. ERROR: Server rejected the 1 private key(s) for jenkins (credentialId:e85865a0-e6eb-4de3-a77e-b0fbbc9ec8c4/method:publickey) [08/17/17 10:46:28] [SSH] Authentication failed. Authentication failed.   So it seems like the "Manually provided key" method will work for me, but there is an issue with key mismatch. So, I'll follow up with our sys admin.
            Hide
            mc1arke Michael Clarke added a comment -

            M Chon Which id_rsa.pub was it you copied in to the configuration? Your personal one (~/.ssh/id_rsa.pub), or the one for the host's public key (/etc/sshd/id_rsa.pub or similar)? It needs to be the latter for this to work.

            Your final failure indicates that your private key (used for authentication) does not match any public key saved on the host, which no amount of key verification is going to fix.

            Using Non verifying Verification Strategy is strongly discouraged, unless you can verify the identity of the host through another means. Have you considered using the "Manually Trusted Key Verification Strategy" to retrieve the remote host's key and then for you to verify the key against the contents of rsa_id.pub from the remote host? This should get round any delays with getting System Admins to perform actions on your behalf.

            Show
            mc1arke Michael Clarke added a comment - M Chon Which id_rsa.pub was it you copied in to the configuration? Your personal one (~/.ssh/id_rsa.pub), or the one for the host's public key (/etc/sshd/id_rsa.pub or similar)? It needs to be the latter for this to work. Your final failure indicates that your private key (used for authentication) does not match any public key saved on the host, which no amount of key verification is going to fix. Using Non verifying Verification Strategy is strongly discouraged , unless you can verify the identity of the host through another means. Have you considered using the "Manually Trusted Key Verification Strategy" to retrieve the remote host's key and then for you to verify the key against the contents of rsa_id.pub from the remote host? This should get round any delays with getting System Admins to perform actions on your behalf.

              People

              • Assignee:
                mc1arke Michael Clarke
                Reporter:
                wl2776 Vladimir Eremeev
              • Votes:
                55 Vote for this issue
                Watchers:
                77 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: