Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-42959

Failed known_hosts verification for SSH agent

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: ssh-slaves-plugin
    • Labels:
    • Environment:
      Versions:

      Jenkins: 2.51
      SSH Agent Plugin: 1.14
      SSH Credentials plugin: 1.13
      SSH Slaves Plugin : 1.15

      Ubuntu 14.04, 16.04
    • Similar Issues:

      Description

      SSH agent isn't launched after the latest update, complaining about missing records in the known hosts file.

      Nevertheless, the records do exist. I've tried to connect manually with ssh, everything was fine, ssh reports that it has found the host in known_hosts.

      Probable reason is the hashed host name in known_hosts.

      Here is the log from the Jenkins slave launch page:

      [03/21/17 11:40:34] [SSH] Opening SSH connection to xxx.xxx.xxx.xxx:22
      [03/21/17 11:40:34] [SSH] WARNING: No entry currently exists in the Known Hosts file for this host.      Connections will be denied until this new host and its associated key is added to the Known Hosts file.
           Key exchange was not finished, connection is closed.
           java.io.IOException: There was a problem while connecting to xxx.xxx.xxx.xxx:22
      	at com.trilead.ssh2.Connection.connect(Connection.java:818)
      	at com.trilead.ssh2.Connection.connect(Connection.java:687)
      	at com.trilead.ssh2.Connection.connect(Connection.java:601)
      	at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:1265)
      	at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:790)
      	at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:785)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
           Caused by: java.io.IOException: Key exchange was not finished, connection is closed.
      	at com.trilead.ssh2.transport.KexManager.getOrWaitForConnectionInfo(KexManager.java:93)
      	at com.trilead.ssh2.transport.TransportManager.getConnectionInfo(TransportManager.java:230)
      	at com.trilead.ssh2.Connection.connect(Connection.java:770)
      	... 9 more
           Caused by: java.io.IOException: The server hostkey was not accepted by the verifier callback
      	at com.trilead.ssh2.transport.KexManager.handleMessage(KexManager.java:591)
      	at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:777)
      	at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:489)
      	... 1 more
           [03/21/17 11:40:34] Launch failed - cleaning up connection
           [03/21/17 11:40:34] [SSH] Connection closed.
      
      

      And this is the debug output from ssh, showing that it has found a record in the second line of known_hosts.

        $ sudo -u jenkins -g jenkins ssh -v jenkins@xxx.xxx.xxx.xxx
          OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
          debug1: Reading configuration data /etc/ssh/ssh_config
          debug1: /etc/ssh/ssh_config line 19: Applying options for *
          debug1: Connecting to xxx.xxx.xxx.xxx [xxx.xxx.xxx.xxx] port 22.
          debug1: Connection established.
          debug1: identity file /var/lib/jenkins/.ssh/id_rsa type 1
      ...
          debug1: sending SSH2_MSG_KEX_ECDH_INIT
          debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
          debug1: Server host key: ECDSA xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx
          debug1: Host 'xxx.xxx.xxx.xxx.xxx' is known and matches the ECDSA host key.
          debug1: Found key in /var/lib/jenkins/.ssh/known_hosts:2
          debug1: ssh_ecdsa_verify: signature correct
      ...
          Welcome to Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-67-generic x86_64)
      

      Additional logs from Jenkins system log:

      Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
          kex_algo=diffie-hellman-group14-sha1
      Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
          server_host_key_algo=ssh-rsa
          Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
          enc_algo_client_to_server=aes256-ctr
          Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
          enc_algo_server_to_client=aes256-ctr
          Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
          mac_algo_client_to_server=hmac-sha1
          Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
          mac_algo_server_to_client=hmac-sha1
          Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
          comp_algo_client_to_server=none
          Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
          comp_algo_server_to_client=none
          Mar 21, 2017 12:06:08 PM FINE com.trilead.ssh2.transport.TransportManager
          Receive thread: error in receiveLoop
          java.io.IOException: The server hostkey was not accepted by the verifier callback
                  at com.trilead.ssh2.transport.KexManager.handleMessage(KexManager.java:591)
                  at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:777)
                  at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:489)
                  at java.lang.Thread.run(Thread.java:745)
      
          Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.TransportManager
          Receive thread: back from receiveLoop
          Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
          kex_algo=diffie-hellman-group14-sha1
          Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
          server_host_key_algo=ssh-rsa
          Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
          enc_algo_client_to_server=aes256-ctr
          Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
          enc_algo_server_to_client=aes256-ctr
          Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
          mac_algo_client_to_server=hmac-sha1
          Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
          mac_algo_server_to_client=hmac-sha1
          Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
          comp_algo_client_to_server=none
          Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
          comp_algo_server_to_client=none
          Mar 21, 2017 12:06:10 PM FINE com.trilead.ssh2.transport.TransportManager
          Receive thread: error in receiveLoop
          java.io.IOException: The server hostkey was not accepted by the verifier callback
                  at com.trilead.ssh2.transport.KexManager.handleMessage(KexManager.java:591)
                  at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:777)
                  at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:489)
                  at java.lang.Thread.run(Thread.java:745)
      
          Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.TransportManager
          Receive thread: back from receiveLoop
      

      File /var/lib/jenkins/.ssh/known_hosts contains strings, looking like being base64-encoded, delimited by '|'. Here is the sample.

      |1|DAg  ...   o... 1ll9wI=| ...  ....  tIrM= ecdsa-sha2-nistp256 xxxxxx..... bmlzdHAyNTYAAAAIbm................. .................xxxxxxxxxxxoKEHF3Vr0q685jI2+6vWjvAAG4lz5Ckujy9k=
      

      Github issue

        Attachments

          Issue Links

            Activity

            wl2776 Vladimir Eremeev created issue -
            wl2776 Vladimir Eremeev made changes -
            Field Original Value New Value
            Description SSH agent isn't launched after the latest update, complaining about missing records in the known hosts file.

            Nevertheless, the records do exist. I've tried to connect manually with ssh, everything was fine, ssh reports that it has found the host in known_hosts.

            Probable reason is the hashed host name in known_hosts.

            Here is the log from the Jenkins slave launch page:

            {noformat}
            [03/21/17 11:40:34] [SSH] Opening SSH connection to xxx.xxx.xxx.xxx:22
            [03/21/17 11:40:34] [SSH] WARNING: No entry currently exists in the Known Hosts file for this host. Connections will be denied until this new host and its associated key is added to the Known Hosts file.
                 Key exchange was not finished, connection is closed.
                 java.io.IOException: There was a problem while connecting to xxx.xxx.xxx.xxx:22
            at com.trilead.ssh2.Connection.connect(Connection.java:818)
            at com.trilead.ssh2.Connection.connect(Connection.java:687)
            at com.trilead.ssh2.Connection.connect(Connection.java:601)
            at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:1265)
            at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:790)
            at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:785)
            at java.util.concurrent.FutureTask.run(FutureTask.java:262)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)
                 Caused by: java.io.IOException: Key exchange was not finished, connection is closed.
            at com.trilead.ssh2.transport.KexManager.getOrWaitForConnectionInfo(KexManager.java:93)
            at com.trilead.ssh2.transport.TransportManager.getConnectionInfo(TransportManager.java:230)
            at com.trilead.ssh2.Connection.connect(Connection.java:770)
            ... 9 more
                 Caused by: java.io.IOException: The server hostkey was not accepted by the verifier callback
            at com.trilead.ssh2.transport.KexManager.handleMessage(KexManager.java:591)
            at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:777)
            at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:489)
            ... 1 more
                 [03/21/17 11:40:34] Launch failed - cleaning up connection
                 [03/21/17 11:40:34] [SSH] Connection closed.

            {noformat}
            And this is the debug output from ssh, showing that it has found a record in the seknown_hosts.
            {noformat}
              $ sudo -u jenkins -g jenkins ssh -v jenkins@xxx.xxx.xxx.xxx
                OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
                debug1: Reading configuration data /etc/ssh/ssh_config
                debug1: /etc/ssh/ssh_config line 19: Applying options for *
                debug1: Connecting to xxx.xxx.xxx.xxx [xxx.xxx.xxx.xxx] port 22.
                debug1: Connection established.
                debug1: identity file /var/lib/jenkins/.ssh/id_rsa type 1
            ...
                debug1: sending SSH2_MSG_KEX_ECDH_INIT
                debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
                debug1: Server host key: ECDSA xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx
                debug1: Host 'xxx.xxx.xxx.xxx.xxx' is known and matches the ECDSA host key.
                debug1: Found key in /var/lib/jenkins/.ssh/known_hosts:2
                debug1: ssh_ecdsa_verify: signature correct
            ...
                Welcome to Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-67-generic x86_64)
            {noformat}

            Additional logs from Jenkins system log:

            {noformat}
            Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
                kex_algo=diffie-hellman-group14-sha1
            Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
                server_host_key_algo=ssh-rsa
                Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
                enc_algo_client_to_server=aes256-ctr
                Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
                enc_algo_server_to_client=aes256-ctr
                Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
                mac_algo_client_to_server=hmac-sha1
                Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
                mac_algo_server_to_client=hmac-sha1
                Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
                comp_algo_client_to_server=none
                Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
                comp_algo_server_to_client=none
                Mar 21, 2017 12:06:08 PM FINE com.trilead.ssh2.transport.TransportManager
                Receive thread: error in receiveLoop
                java.io.IOException: The server hostkey was not accepted by the verifier callback
                        at com.trilead.ssh2.transport.KexManager.handleMessage(KexManager.java:591)
                        at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:777)
                        at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:489)
                        at java.lang.Thread.run(Thread.java:745)

                Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.TransportManager
                Receive thread: back from receiveLoop
                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
                kex_algo=diffie-hellman-group14-sha1
                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
                server_host_key_algo=ssh-rsa
                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
                enc_algo_client_to_server=aes256-ctr
                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
                enc_algo_server_to_client=aes256-ctr
                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
                mac_algo_client_to_server=hmac-sha1
                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
                mac_algo_server_to_client=hmac-sha1
                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
                comp_algo_client_to_server=none
                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
                comp_algo_server_to_client=none
                Mar 21, 2017 12:06:10 PM FINE com.trilead.ssh2.transport.TransportManager
                Receive thread: error in receiveLoop
                java.io.IOException: The server hostkey was not accepted by the verifier callback
                        at com.trilead.ssh2.transport.KexManager.handleMessage(KexManager.java:591)
                        at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:777)
                        at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:489)
                        at java.lang.Thread.run(Thread.java:745)

                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.TransportManager
                Receive thread: back from receiveLoop
            {noformat}

            File /var/lib/jenkins/.ssh/known_hosts contains strings, looking like being base64-encoded, delimited by '|'. Here is the sample.
            {noformat}
            |1|DAg ... o... 1ll9wI=| ... .... tIrM= ecdsa-sha2-nistp256 xxxxxx..... bmlzdHAyNTYAAAAIbm................. .................xxxxxxxxxxxoKEHF3Vr0q685jI2+6vWjvAAG4lz5Ckujy9k=
            {noformat}

            [Github issue|github.com/jenkinsci/ssh-slaves-plugin/issues/48]
            SSH agent isn't launched after the latest update, complaining about missing records in the known hosts file.

            Nevertheless, the records do exist. I've tried to connect manually with ssh, everything was fine, ssh reports that it has found the host in known_hosts.

            Probable reason is the hashed host name in known_hosts.

            Here is the log from the Jenkins slave launch page:

            {noformat}
            [03/21/17 11:40:34] [SSH] Opening SSH connection to xxx.xxx.xxx.xxx:22
            [03/21/17 11:40:34] [SSH] WARNING: No entry currently exists in the Known Hosts file for this host. Connections will be denied until this new host and its associated key is added to the Known Hosts file.
                 Key exchange was not finished, connection is closed.
                 java.io.IOException: There was a problem while connecting to xxx.xxx.xxx.xxx:22
            at com.trilead.ssh2.Connection.connect(Connection.java:818)
            at com.trilead.ssh2.Connection.connect(Connection.java:687)
            at com.trilead.ssh2.Connection.connect(Connection.java:601)
            at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:1265)
            at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:790)
            at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:785)
            at java.util.concurrent.FutureTask.run(FutureTask.java:262)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)
                 Caused by: java.io.IOException: Key exchange was not finished, connection is closed.
            at com.trilead.ssh2.transport.KexManager.getOrWaitForConnectionInfo(KexManager.java:93)
            at com.trilead.ssh2.transport.TransportManager.getConnectionInfo(TransportManager.java:230)
            at com.trilead.ssh2.Connection.connect(Connection.java:770)
            ... 9 more
                 Caused by: java.io.IOException: The server hostkey was not accepted by the verifier callback
            at com.trilead.ssh2.transport.KexManager.handleMessage(KexManager.java:591)
            at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:777)
            at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:489)
            ... 1 more
                 [03/21/17 11:40:34] Launch failed - cleaning up connection
                 [03/21/17 11:40:34] [SSH] Connection closed.

            {noformat}
            And this is the debug output from ssh, showing that it has found a record in the seknown_hosts.
            {noformat}
              $ sudo -u jenkins -g jenkins ssh -v jenkins@xxx.xxx.xxx.xxx
                OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
                debug1: Reading configuration data /etc/ssh/ssh_config
                debug1: /etc/ssh/ssh_config line 19: Applying options for *
                debug1: Connecting to xxx.xxx.xxx.xxx [xxx.xxx.xxx.xxx] port 22.
                debug1: Connection established.
                debug1: identity file /var/lib/jenkins/.ssh/id_rsa type 1
            ...
                debug1: sending SSH2_MSG_KEX_ECDH_INIT
                debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
                debug1: Server host key: ECDSA xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx
                debug1: Host 'xxx.xxx.xxx.xxx.xxx' is known and matches the ECDSA host key.
                debug1: Found key in /var/lib/jenkins/.ssh/known_hosts:2
                debug1: ssh_ecdsa_verify: signature correct
            ...
                Welcome to Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-67-generic x86_64)
            {noformat}

            Additional logs from Jenkins system log:

            {noformat}
            Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
                kex_algo=diffie-hellman-group14-sha1
            Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
                server_host_key_algo=ssh-rsa
                Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
                enc_algo_client_to_server=aes256-ctr
                Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
                enc_algo_server_to_client=aes256-ctr
                Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
                mac_algo_client_to_server=hmac-sha1
                Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
                mac_algo_server_to_client=hmac-sha1
                Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
                comp_algo_client_to_server=none
                Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
                comp_algo_server_to_client=none
                Mar 21, 2017 12:06:08 PM FINE com.trilead.ssh2.transport.TransportManager
                Receive thread: error in receiveLoop
                java.io.IOException: The server hostkey was not accepted by the verifier callback
                        at com.trilead.ssh2.transport.KexManager.handleMessage(KexManager.java:591)
                        at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:777)
                        at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:489)
                        at java.lang.Thread.run(Thread.java:745)

                Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.TransportManager
                Receive thread: back from receiveLoop
                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
                kex_algo=diffie-hellman-group14-sha1
                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
                server_host_key_algo=ssh-rsa
                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
                enc_algo_client_to_server=aes256-ctr
                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
                enc_algo_server_to_client=aes256-ctr
                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
                mac_algo_client_to_server=hmac-sha1
                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
                mac_algo_server_to_client=hmac-sha1
                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
                comp_algo_client_to_server=none
                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
                comp_algo_server_to_client=none
                Mar 21, 2017 12:06:10 PM FINE com.trilead.ssh2.transport.TransportManager
                Receive thread: error in receiveLoop
                java.io.IOException: The server hostkey was not accepted by the verifier callback
                        at com.trilead.ssh2.transport.KexManager.handleMessage(KexManager.java:591)
                        at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:777)
                        at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:489)
                        at java.lang.Thread.run(Thread.java:745)

                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.TransportManager
                Receive thread: back from receiveLoop
            {noformat}

            File /var/lib/jenkins/.ssh/known_hosts contains strings, looking like being base64-encoded, delimited by '|'. Here is the sample.
            {noformat}
            |1|DAg ... o... 1ll9wI=| ... .... tIrM= ecdsa-sha2-nistp256 xxxxxx..... bmlzdHAyNTYAAAAIbm................. .................xxxxxxxxxxxoKEHF3Vr0q685jI2+6vWjvAAG4lz5Ckujy9k=
            {noformat}

            [Github issue|https://github.com/jenkinsci/ssh-slaves-plugin/issues/48]
            wl2776 Vladimir Eremeev made changes -
            Description SSH agent isn't launched after the latest update, complaining about missing records in the known hosts file.

            Nevertheless, the records do exist. I've tried to connect manually with ssh, everything was fine, ssh reports that it has found the host in known_hosts.

            Probable reason is the hashed host name in known_hosts.

            Here is the log from the Jenkins slave launch page:

            {noformat}
            [03/21/17 11:40:34] [SSH] Opening SSH connection to xxx.xxx.xxx.xxx:22
            [03/21/17 11:40:34] [SSH] WARNING: No entry currently exists in the Known Hosts file for this host. Connections will be denied until this new host and its associated key is added to the Known Hosts file.
                 Key exchange was not finished, connection is closed.
                 java.io.IOException: There was a problem while connecting to xxx.xxx.xxx.xxx:22
            at com.trilead.ssh2.Connection.connect(Connection.java:818)
            at com.trilead.ssh2.Connection.connect(Connection.java:687)
            at com.trilead.ssh2.Connection.connect(Connection.java:601)
            at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:1265)
            at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:790)
            at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:785)
            at java.util.concurrent.FutureTask.run(FutureTask.java:262)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)
                 Caused by: java.io.IOException: Key exchange was not finished, connection is closed.
            at com.trilead.ssh2.transport.KexManager.getOrWaitForConnectionInfo(KexManager.java:93)
            at com.trilead.ssh2.transport.TransportManager.getConnectionInfo(TransportManager.java:230)
            at com.trilead.ssh2.Connection.connect(Connection.java:770)
            ... 9 more
                 Caused by: java.io.IOException: The server hostkey was not accepted by the verifier callback
            at com.trilead.ssh2.transport.KexManager.handleMessage(KexManager.java:591)
            at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:777)
            at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:489)
            ... 1 more
                 [03/21/17 11:40:34] Launch failed - cleaning up connection
                 [03/21/17 11:40:34] [SSH] Connection closed.

            {noformat}
            And this is the debug output from ssh, showing that it has found a record in the seknown_hosts.
            {noformat}
              $ sudo -u jenkins -g jenkins ssh -v jenkins@xxx.xxx.xxx.xxx
                OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
                debug1: Reading configuration data /etc/ssh/ssh_config
                debug1: /etc/ssh/ssh_config line 19: Applying options for *
                debug1: Connecting to xxx.xxx.xxx.xxx [xxx.xxx.xxx.xxx] port 22.
                debug1: Connection established.
                debug1: identity file /var/lib/jenkins/.ssh/id_rsa type 1
            ...
                debug1: sending SSH2_MSG_KEX_ECDH_INIT
                debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
                debug1: Server host key: ECDSA xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx
                debug1: Host 'xxx.xxx.xxx.xxx.xxx' is known and matches the ECDSA host key.
                debug1: Found key in /var/lib/jenkins/.ssh/known_hosts:2
                debug1: ssh_ecdsa_verify: signature correct
            ...
                Welcome to Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-67-generic x86_64)
            {noformat}

            Additional logs from Jenkins system log:

            {noformat}
            Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
                kex_algo=diffie-hellman-group14-sha1
            Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
                server_host_key_algo=ssh-rsa
                Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
                enc_algo_client_to_server=aes256-ctr
                Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
                enc_algo_server_to_client=aes256-ctr
                Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
                mac_algo_client_to_server=hmac-sha1
                Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
                mac_algo_server_to_client=hmac-sha1
                Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
                comp_algo_client_to_server=none
                Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
                comp_algo_server_to_client=none
                Mar 21, 2017 12:06:08 PM FINE com.trilead.ssh2.transport.TransportManager
                Receive thread: error in receiveLoop
                java.io.IOException: The server hostkey was not accepted by the verifier callback
                        at com.trilead.ssh2.transport.KexManager.handleMessage(KexManager.java:591)
                        at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:777)
                        at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:489)
                        at java.lang.Thread.run(Thread.java:745)

                Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.TransportManager
                Receive thread: back from receiveLoop
                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
                kex_algo=diffie-hellman-group14-sha1
                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
                server_host_key_algo=ssh-rsa
                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
                enc_algo_client_to_server=aes256-ctr
                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
                enc_algo_server_to_client=aes256-ctr
                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
                mac_algo_client_to_server=hmac-sha1
                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
                mac_algo_server_to_client=hmac-sha1
                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
                comp_algo_client_to_server=none
                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
                comp_algo_server_to_client=none
                Mar 21, 2017 12:06:10 PM FINE com.trilead.ssh2.transport.TransportManager
                Receive thread: error in receiveLoop
                java.io.IOException: The server hostkey was not accepted by the verifier callback
                        at com.trilead.ssh2.transport.KexManager.handleMessage(KexManager.java:591)
                        at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:777)
                        at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:489)
                        at java.lang.Thread.run(Thread.java:745)

                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.TransportManager
                Receive thread: back from receiveLoop
            {noformat}

            File /var/lib/jenkins/.ssh/known_hosts contains strings, looking like being base64-encoded, delimited by '|'. Here is the sample.
            {noformat}
            |1|DAg ... o... 1ll9wI=| ... .... tIrM= ecdsa-sha2-nistp256 xxxxxx..... bmlzdHAyNTYAAAAIbm................. .................xxxxxxxxxxxoKEHF3Vr0q685jI2+6vWjvAAG4lz5Ckujy9k=
            {noformat}

            [Github issue|https://github.com/jenkinsci/ssh-slaves-plugin/issues/48]
            SSH agent isn't launched after the latest update, complaining about missing records in the known hosts file.

            Nevertheless, the records do exist. I've tried to connect manually with ssh, everything was fine, ssh reports that it has found the host in known_hosts.

            Probable reason is the hashed host name in known_hosts.

            Here is the log from the Jenkins slave launch page:

            {noformat}
            [03/21/17 11:40:34] [SSH] Opening SSH connection to xxx.xxx.xxx.xxx:22
            [03/21/17 11:40:34] [SSH] WARNING: No entry currently exists in the Known Hosts file for this host. Connections will be denied until this new host and its associated key is added to the Known Hosts file.
                 Key exchange was not finished, connection is closed.
                 java.io.IOException: There was a problem while connecting to xxx.xxx.xxx.xxx:22
            at com.trilead.ssh2.Connection.connect(Connection.java:818)
            at com.trilead.ssh2.Connection.connect(Connection.java:687)
            at com.trilead.ssh2.Connection.connect(Connection.java:601)
            at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:1265)
            at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:790)
            at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:785)
            at java.util.concurrent.FutureTask.run(FutureTask.java:262)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)
                 Caused by: java.io.IOException: Key exchange was not finished, connection is closed.
            at com.trilead.ssh2.transport.KexManager.getOrWaitForConnectionInfo(KexManager.java:93)
            at com.trilead.ssh2.transport.TransportManager.getConnectionInfo(TransportManager.java:230)
            at com.trilead.ssh2.Connection.connect(Connection.java:770)
            ... 9 more
                 Caused by: java.io.IOException: The server hostkey was not accepted by the verifier callback
            at com.trilead.ssh2.transport.KexManager.handleMessage(KexManager.java:591)
            at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:777)
            at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:489)
            ... 1 more
                 [03/21/17 11:40:34] Launch failed - cleaning up connection
                 [03/21/17 11:40:34] [SSH] Connection closed.

            {noformat}
            And this is the debug output from ssh, showing that it has found a record in the second line of known_hosts.
            {noformat}
              $ sudo -u jenkins -g jenkins ssh -v jenkins@xxx.xxx.xxx.xxx
                OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
                debug1: Reading configuration data /etc/ssh/ssh_config
                debug1: /etc/ssh/ssh_config line 19: Applying options for *
                debug1: Connecting to xxx.xxx.xxx.xxx [xxx.xxx.xxx.xxx] port 22.
                debug1: Connection established.
                debug1: identity file /var/lib/jenkins/.ssh/id_rsa type 1
            ...
                debug1: sending SSH2_MSG_KEX_ECDH_INIT
                debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
                debug1: Server host key: ECDSA xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx
                debug1: Host 'xxx.xxx.xxx.xxx.xxx' is known and matches the ECDSA host key.
                debug1: Found key in /var/lib/jenkins/.ssh/known_hosts:2
                debug1: ssh_ecdsa_verify: signature correct
            ...
                Welcome to Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-67-generic x86_64)
            {noformat}

            Additional logs from Jenkins system log:

            {noformat}
            Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
                kex_algo=diffie-hellman-group14-sha1
            Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
                server_host_key_algo=ssh-rsa
                Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
                enc_algo_client_to_server=aes256-ctr
                Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
                enc_algo_server_to_client=aes256-ctr
                Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
                mac_algo_client_to_server=hmac-sha1
                Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
                mac_algo_server_to_client=hmac-sha1
                Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
                comp_algo_client_to_server=none
                Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
                comp_algo_server_to_client=none
                Mar 21, 2017 12:06:08 PM FINE com.trilead.ssh2.transport.TransportManager
                Receive thread: error in receiveLoop
                java.io.IOException: The server hostkey was not accepted by the verifier callback
                        at com.trilead.ssh2.transport.KexManager.handleMessage(KexManager.java:591)
                        at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:777)
                        at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:489)
                        at java.lang.Thread.run(Thread.java:745)

                Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.TransportManager
                Receive thread: back from receiveLoop
                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
                kex_algo=diffie-hellman-group14-sha1
                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
                server_host_key_algo=ssh-rsa
                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
                enc_algo_client_to_server=aes256-ctr
                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
                enc_algo_server_to_client=aes256-ctr
                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
                mac_algo_client_to_server=hmac-sha1
                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
                mac_algo_server_to_client=hmac-sha1
                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
                comp_algo_client_to_server=none
                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
                comp_algo_server_to_client=none
                Mar 21, 2017 12:06:10 PM FINE com.trilead.ssh2.transport.TransportManager
                Receive thread: error in receiveLoop
                java.io.IOException: The server hostkey was not accepted by the verifier callback
                        at com.trilead.ssh2.transport.KexManager.handleMessage(KexManager.java:591)
                        at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:777)
                        at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:489)
                        at java.lang.Thread.run(Thread.java:745)

                Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.TransportManager
                Receive thread: back from receiveLoop
            {noformat}

            File /var/lib/jenkins/.ssh/known_hosts contains strings, looking like being base64-encoded, delimited by '|'. Here is the sample.
            {noformat}
            |1|DAg ... o... 1ll9wI=| ... .... tIrM= ecdsa-sha2-nistp256 xxxxxx..... bmlzdHAyNTYAAAAIbm................. .................xxxxxxxxxxxoKEHF3Vr0q685jI2+6vWjvAAG4lz5Ckujy9k=
            {noformat}

            [Github issue|https://github.com/jenkinsci/ssh-slaves-plugin/issues/48]
            jglick Jesse Glick made changes -
            Component/s ssh-credentials-plugin [ 17424 ]
            jglick Jesse Glick made changes -
            Assignee Stephen Connolly [ stephenconnolly ] Michael Clarke [ mc1arke ]
            jglick Jesse Glick made changes -
            Labels regression
            jglick Jesse Glick made changes -
            Link This issue depends on SECURITY-161 [ SECURITY-161 ]
            wl2776 Vladimir Eremeev made changes -
            Attachment J42959.png [ 36714 ]
            mc1arke Michael Clarke made changes -
            Remote Link This issue links to "Add support for Ed25519 host and user keys (Web Link)" [ 15844 ]
            mc1arke Michael Clarke made changes -
            Remote Link This issue links to "Add support for host and user ECDSA keys (Web Link)" [ 15845 ]
            mc1arke Michael Clarke made changes -
            Remote Link This issue links to "Prefer algorithms currently supported from known_hosts entries (Web Link)" [ 15846 ]
            mc1arke Michael Clarke made changes -
            Status Open [ 1 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            dmazuronak Dzianis Mazuronak made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            moshe_zvi Moshe Zvi made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            mc1arke Michael Clarke made changes -
            Status Reopened [ 4 ] In Progress [ 3 ]
            mc1arke Michael Clarke made changes -
            Remote Link This issue links to "Specify preferred host keys during connect (Web Link)" [ 16804 ]
            markewaite Mark Waite made changes -
            Link This issue is duplicated by JENKINS-44803 [ JENKINS-44803 ]
            scm_issue_link SCM/JIRA link daemon made changes -
            Status In Progress [ 3 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            rtyler R. Tyler Croy made changes -
            Link This issue relates to JENKINS-44830 [ JENKINS-44830 ]
            jglick Jesse Glick made changes -
            Link This issue is blocked by JENKINS-44832 [ JENKINS-44832 ]
            olenads Olena Gaydarenko made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            mc1arke Michael Clarke made changes -
            Status Reopened [ 4 ] Closed [ 6 ]
            Resolution Fixed [ 1 ]

              People

              • Assignee:
                mc1arke Michael Clarke
                Reporter:
                wl2776 Vladimir Eremeev
              • Votes:
                55 Vote for this issue
                Watchers:
                76 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: