Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Duplicate
    • Labels:
      None
    • Environment:
      Jenkins 1.138.3
      ssh-slaves-plugin 1.29.1
      Java 1.8.0_171-8u171
      OS: Debian Stretch (Server and Jenkins Nodes)
    • Similar Issues:
    • Released As:
      ssh-slaves-1.29.2

      Description

      Starting with ssh-slaves-plugin 1.29.1, the SSH agent connection with all nodes fail.

      Logs on Jenkins (logfile):

      [11/21/18 09:38:38] SSH Launch of HOSTNAME on HOSTNAME failed in 391 ms
      

      Logs on Jenkins (UI):

      SSHLauncher{host='HOSTNAME', port=22, credentialsId='b1f94972-dfce-45d0-9cb1-f14e3dc324b9', jvmOptions='', javaPath='', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=210, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.ManuallyTrustedKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true}
      [11/21/18 09:40:05] [SSH] Opening SSH connection to HOSTNAME:22.
      [11/21/18 09:40:05] [SSH] SSH host key matches key seen previously for this host. Connection will be allowed.
      [11/21/18 09:40:05] [SSH] Authentication failed.
      Authentication failed.
      [11/21/18 09:40:05] Launch failed - cleaning up connection
      [11/21/18 09:40:05] [SSH] Connection closed.
      

      The auth.log of a node just contains:

      Closed due to user request. [preauth]
      

      Downgrading to 1.28.1 fixes the issue.

        Attachments

        1. config.xml
          2 kB
        2. jenkins1.png
          jenkins1.png
          19 kB
        3. jenkins2.png
          jenkins2.png
          143 kB

          Issue Links

            Activity

            Hide
            ifernandezcalvo Ivan Fernandez Calvo added a comment -

            I have tested some configuration to try to replicate the issue, I could not. These are the step I made:

            • Start vanilla Jenkins core 2.157 + ssh-slaves 1.26 + ssh-credential 1.12
            • Create 3 SSH credentials "From the Jenkins master ~/.ssh", "From a file on Jenkins master", and "Enter directly"
            • Create 3 SSH agents using the credential created before
            • Check that every agent connect
            • Upgrade to ssh-slaves 1.29.4 + ssh-credential 1.14 (it is a required dependency)
            • Restart Jenkins
            • Check every agent connect
            • Check every credential is migrated to use DirectEntryPrivateKeySource ("Enter directly")

            I check the code of ssh-credentials, all the credential source type have the proper method readResolve to migrate to "Enter directly", so there is something I do not see. In any case, the issue is related to ssh-credentials and a deprecated type of credentials, the workaround it is to recreate the credential with the same ID using "Enter directly" for the key, probably if you only save again the credential it will be migrated.

            I will put a note in the troubleshooting guide and on changelog breaking changes.

            Show
            ifernandezcalvo Ivan Fernandez Calvo added a comment - I have tested some configuration to try to replicate the issue, I could not. These are the step I made: Start vanilla Jenkins core 2.157 + ssh-slaves 1.26 + ssh-credential 1.12 Create 3 SSH credentials "From the Jenkins master ~/.ssh", "From a file on Jenkins master", and "Enter directly" Create 3 SSH agents using the credential created before Check that every agent connect Upgrade to ssh-slaves 1.29.4 + ssh-credential 1.14 (it is a required dependency) Restart Jenkins Check every agent connect Check every credential is migrated to use DirectEntryPrivateKeySource ("Enter directly") I check the code of ssh-credentials, all the credential source type have the proper method readResolve to migrate to "Enter directly", so there is something I do not see. In any case, the issue is related to ssh-credentials and a deprecated type of credentials, the workaround it is to recreate the credential with the same ID using "Enter directly" for the key, probably if you only save again the credential it will be migrated. I will put a note in the troubleshooting guide and on changelog breaking changes.
            Hide
            sintbert Beat Guggisberg added a comment -

            I had the same problem today after a server reboot.

            Plugin was on 1.29.1 from an earlier update, but the server never rebooted since.

            Now it was not able to connect to the slaves and i got the Old Data warning.

             

            I was able solve the problem by updating all plugins to new version (Slaves plugin upgraded to 1.29.4). Then downgrading the SSH Slaves plugin back to 1.29.1 and after a reboot again upgrading it to 1.29.4.

            Now the slaves work again.

             

            Show
            sintbert Beat Guggisberg added a comment - I had the same problem today after a server reboot. Plugin was on 1.29.1 from an earlier update, but the server never rebooted since. Now it was not able to connect to the slaves and i got the Old Data warning.   I was able solve the problem by updating all plugins to new version (Slaves plugin upgraded to 1.29.4). Then downgrading the SSH Slaves plugin back to 1.29.1 and after a reboot again upgrading it to 1.29.4. Now the slaves work again.  
            Hide
            dnusbaum Devin Nusbaum added a comment -

            I think this is a dupe of JENKINS-52232. The exception from the old data monitor posted here gave me an idea of a possible mitigation, although the fact that that code is running as anonymous at all is a big question and is probably the root cause.

            Show
            dnusbaum Devin Nusbaum added a comment - I think this is a dupe of JENKINS-52232 . The exception from the old data monitor posted here gave me an idea of a possible mitigation, although the fact that that code is running as anonymous at all is a big question and is probably the root cause.
            Hide
            ifernandezcalvo Ivan Fernandez Calvo added a comment -

            I'll close this one, then we continue in JENKINS-52232.

            Show
            ifernandezcalvo Ivan Fernandez Calvo added a comment - I'll close this one, then we continue in JENKINS-52232 .
            Hide
            vazhnov Alexey Vazhnov added a comment -

            From SECURITY-440 / CVE-2018-1000601 fix:

            SSH Credentials Plugin no longer supports SSH credentials from files on the Jenkins master file system, neither user-specified file paths nor ~/.ssh. Existing SSH credentials of these kinds are migrated to "directly entered" SSH credentials.

            Show
            vazhnov Alexey Vazhnov added a comment - From SECURITY-440 / CVE-2018-1000601 fix: SSH Credentials Plugin no longer supports SSH credentials from files on the Jenkins master file system, neither user-specified file paths nor ~/.ssh. Existing SSH credentials of these kinds are migrated to "directly entered" SSH credentials.

              People

              • Assignee:
                ifernandezcalvo Ivan Fernandez Calvo
                Reporter:
                notizblock Florian P
              • Votes:
                9 Vote for this issue
                Watchers:
                19 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: