Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48413

Hosts unreachable when using a private key with passphrase provided using the Credentials plugin

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Blocker
    • Resolution: Unresolved
    • Component/s: ansible-plugin
    • Labels:
      None
    • Environment:
      Debian Jessie
      OpenSSH_6.7p1 Debian-5+deb8u3, OpenSSL 1.0.1t 3 May 2016
      jenkins 2.73.3 (stable)
      with plugin ansible 0.6.2
      and plugin credentials 2.1.16
      ansible-playbook 2.4.2.0
    • Similar Issues:

      Description

      TL;DR: it seems the ansible plugin does not get/provide the passphrase correctly from/to the credentials plugin.

      See also: -JENKINS-20879-

      When I run a job with an Ansible build task that uses a private key with a passphrase provided by the Credentials plugin, the playbook hangs.

       

      When I add the

      --ssh-extra-args="-o BatchMode=yes"

      option to the build, Ansible fails quickly, and hosts are unreachable. That indicates to me that SSH is prompting for the passphrase of my private key. As the jenkins job is not interactive, it hangs without the option.

       

      I have also tested the following:

      • the playbook's execution is OK using the passphraseless key
      • i can reach the hosts using a manual SSH command with the passphrase-enabled key, after being prompted for the passphrase by SSH
      • the passphrase stored by the Credentials plugin seems fine : during some tests I could  see a temporary .sh file generated in the $CATALINA_HOME/temp folder of Jenkins/Tomcat, that contains the passphrase in clear-text, and is used to generate a temporary PEM file (.key) containing the deciphered key

      All in all it seems the only remaining explaination is that there is a bug in the implementation of the Ansible plugin.

      The following SSH debug output is generated by Ansible with options :

       --ssh-extra-args="-o BatchMode=yes"

      and

      -vvvvv

       

      debug1: Next authentication method: publickey
      debug1: Trying private key: /usr/local/tomcat/temp/ssh1471148055772625127.key
      debug1: key_load_private_type: incorrect passphrase supplied to decrypt private key
      debug2: we did not send a packet, disable method
      debug1: No more authentication methods to try.
       Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

      But, as I understand the SSH message incorrect passphrase supplied to decrypt private key, it can also mean the PEM file is corrupted. And in fact, when I can see the file it is empty (0 byte).

        Attachments

          Activity

          Hide
          bardelotnzl Noël Bardelot added a comment - - edited

          Note that in order to see the .sh and .key files in $CATALINA_HOME/temp you can remove the option --ssh-extra-args="-o BatchMode=yes" and let the job hang...

          Here is an example of those files:

          -r-------- 1 docker docker  0 déc.   6 18:16 ssh635376564463353267.key
          -rwx------ 1 docker docker 43 déc.   6 18:16 ssh6728840452037021845.sh
          
          $ cat ssh635376564463353267.key && echo ""
          
          $ cat ssh6728840452037021845.sh && echo ""
          #! /bin/sh
          /bin/echo "thepassphrase"
          
          Show
          bardelotnzl Noël Bardelot added a comment - - edited Note that in order to see the .sh and .key files in $CATALINA_HOME/temp you can remove the option --ssh-extra-args="-o BatchMode=yes" and let the job hang... Here is an example of those files: -r-------- 1 docker docker  0 déc.   6 18:16 ssh635376564463353267.key -rwx------ 1 docker docker 43 déc.   6 18:16 ssh6728840452037021845.sh $ cat ssh635376564463353267.key && echo "" $ cat ssh6728840452037021845.sh && echo "" #! /bin/sh /bin/echo "thepassphrase"
          Hide
          bardelotnzl Noël Bardelot added a comment - - edited

          Clearly this code is causing the issue:

              protected ArgumentListBuilder appendCredentials(ArgumentListBuilder args)
                      throws IOException, InterruptedException
              {
                  if (credentials instanceof SSHUserPrivateKey) {
                      SSHUserPrivateKey privateKeyCredentials = (SSHUserPrivateKey)credentials;
                      key = Utils.createSshKeyFile(key, ws, privateKeyCredentials, copyCredentialsInWorkspace);
                      args.add("--private-key").add(key.getRemote().replace("%", "%%"));
                      args.add("-u").add(privateKeyCredentials.getUsername());
                      if (privateKeyCredentials.getPassphrase() != null) {
                          script = Utils.createSshAskPassFile(script, ws, privateKeyCredentials, copyCredentialsInWorkspace);
                          environment.put("SSH_ASKPASS", script.getRemote());
                          // inspired from https://github.com/jenkinsci/git-client-plugin/pull/168
                          // but does not work with MacOSX
                          if (! environment.containsKey("DISPLAY")) {
                              environment.put("DISPLAY", ":123.456");
                          }
                      }
                  } else if (credentials instanceof UsernamePasswordCredentials) {
          ...
                  }
                  return args;
              }
          

          This code is entirely based on the fact that some SSH_ASKPASS environment variable is created that contains the key passphrase. But that environment variable isn't used anywhere else in the plugin. It seems the plugin's trying to copy the behaviour of the Git plugin. Some I'll go take a look at that one to see how they make use of SSH_ASKPASS.

          Show
          bardelotnzl Noël Bardelot added a comment - - edited Clearly this code is causing the issue: protected ArgumentListBuilder appendCredentials(ArgumentListBuilder args) throws IOException, InterruptedException { if (credentials instanceof SSHUserPrivateKey) { SSHUserPrivateKey privateKeyCredentials = (SSHUserPrivateKey)credentials; key = Utils.createSshKeyFile(key, ws, privateKeyCredentials, copyCredentialsInWorkspace); args.add( "-- private -key" ).add(key.getRemote().replace( "%" , "%%" )); args.add( "-u" ).add(privateKeyCredentials.getUsername()); if (privateKeyCredentials.getPassphrase() != null ) { script = Utils.createSshAskPassFile(script, ws, privateKeyCredentials, copyCredentialsInWorkspace); environment.put( "SSH_ASKPASS" , script.getRemote()); // inspired from https://github.com/jenkinsci/git-client-plugin/pull/168 // but does not work with MacOSX if (! environment.containsKey( "DISPLAY" )) { environment.put( "DISPLAY" , ":123.456" ); } } } else if (credentials instanceof UsernamePasswordCredentials) { ... } return args; } This code is entirely based on the fact that some SSH_ASKPASS environment variable is created that contains the key passphrase. But that environment variable isn't used anywhere else in the plugin. It seems the plugin's trying to copy the behaviour of the Git plugin. Some I'll go take a look at that one to see how they make use of SSH_ASKPASS.
          Hide
          bardelotnzl Noël Bardelot added a comment - - edited

          See also this commit at Ansible: https://github.com/quipucords/rho/pull/246/commits/7be602a4a76b5af4ed427e7e32bdf490b3c9c130 on 30 Aug 2017

          that indicates that the supported way to use passphrases with Ansible is by using ssh-agent.

          And also (with the participation of the maintainer of the ansible plugin) : http://jenkins-ci.361315.n4.nabble.com/jenkins-ssh-agent-ansible-td4853087.html

          Adding all this up, I think the nice way to go is to use the prependPasswordCredentials method in order to run/kill the ssh-agent the same way sshpass is used (with the added complexity of having to kill the agent after the ansible command ends).

          Show
          bardelotnzl Noël Bardelot added a comment - - edited See also this commit at Ansible: https://github.com/quipucords/rho/pull/246/commits/7be602a4a76b5af4ed427e7e32bdf490b3c9c130 on 30 Aug 2017 that indicates that the supported way to use passphrases with Ansible is by using ssh-agent. And also (with the participation of the maintainer of the ansible plugin) : http://jenkins-ci.361315.n4.nabble.com/jenkins-ssh-agent-ansible-td4853087.html Adding all this up, I think the nice way to go is to use the prependPasswordCredentials method in order to run/kill the ssh-agent the same way sshpass is used (with the added complexity of having to kill the agent after the ansible command ends).
          Hide
          xleliberty Xavier LEMBO added a comment - - edited

          Hi, 

          I can confirme the same behaviour. 

          with a working playbook, once upgraded to 2.4.0 or 2.4.1 or 2.4.2 , the playbook fail with same ssh-key error.

           

          Thanks

          Show
          xleliberty Xavier LEMBO added a comment - - edited Hi,  I can confirme the same behaviour.  with a working playbook, once upgraded to 2.4.0 or 2.4.1 or 2.4.2 , the playbook fail with same ssh-key error.   Thanks
          Hide
          bardelotnzl Noël Bardelot added a comment - - edited

          After some reasearch I now understand that SSH_ASKPASS is a SSH standard.

          SSH_ASKPASS' If ssh needs a passphrase, it will read the passphrase from the current terminal if it was run from a terminal. If ssh does not have a terminal associated with it but DISPLAY and SSH_ASKPASS are set, it will execute the program specified by SSH_ASKPASS and open an X11 window to read the passphrase. This is particularly useful when calling ssh from a .xsession or related script. (Note that on some machines it may be necessary to redirect the input from /dev/null to make this work.)

          My previous comment about SSH_ASKPASS being set to the passphrase is false, it is set to the path of the script providing the passphrase. Plus, the DISPLAY environment variable is set to a mock value as to match the case described by the SSH documentation :

          if (! environment.containsKey("DISPLAY")) {
              environment.put("DISPLAY", ":123.456");
          }
          
          Show
          bardelotnzl Noël Bardelot added a comment - - edited After some reasearch I now understand that SSH_ASKPASS is a SSH standard. SSH_ASKPASS' If ssh needs a passphrase, it will read the passphrase from the current terminal if it was run from a terminal. If ssh does not have a terminal associated with it but DISPLAY and SSH_ASKPASS are set, it will execute the program specified by SSH_ASKPASS and open an X11 window to read the passphrase. This is particularly useful when calling ssh from a .xsession or related script. (Note that on some machines it may be necessary to redirect the input from /dev/null to make this work.) My previous comment about SSH_ASKPASS being set to the passphrase is false, it is set to the path of the script providing the passphrase. Plus, the DISPLAY environment variable is set to a mock value as to match the case described by the SSH documentation : if (! environment.containsKey( "DISPLAY" )) { environment.put( "DISPLAY" , ":123.456" ); }
          Hide
          bardelotnzl Noël Bardelot added a comment - - edited

          After some tests :

          echo "$SSH_ASKPASS"
          /path/to/ssh_askpass.sh # simply echo "thepassphrase"
          echo "$DISPLAY"
          :123.456
          
          ssh -i key user@host -> prompted "Enter passphrase for key"
          setsid ssh -i key user@host -> Logged in!
          Same without DISPLAY -> SSH prints permission denied (x3)
          
          Show
          bardelotnzl Noël Bardelot added a comment - - edited After some tests : echo "$SSH_ASKPASS" /path/to/ssh_askpass.sh # simply echo "thepassphrase" echo "$DISPLAY" :123.456 ssh -i key user@host -> prompted "Enter passphrase for key" setsid ssh -i key user@host -> Logged in! Same without DISPLAY -> SSH prints permission denied (x3)
          Hide
          bardelotnzl Noël Bardelot added a comment - - edited

          Running this from the command line fails:

          export SSH_ASKPASS="/path/to/ssh_askpass.sh"
          export DISPLAY=":123.456"
          export ANSIBLE_CONFIG=/path/to/ansible.cfg
          ansible-playbook /path/to/some.yml \
                        -i /path/to/hosts \
                        --private-key /path/to/private_key \
                        -u user \
                        --ssh-extra-args="-o BatchMode=yes" \
                        -vvvvv
          

          with:

          debug1: Offering RSA public key: /path/to/private_key
          debug3: send_pubkey_test
          debug2: we sent a publickey packet, wait for reply
          debug1: Server accepts key: pkalg ssh-rsa blen 279
          debug2: input_userauth_pk_ok: ...
          debug3: sign_and_send_pubkey: RSA ...
          debug1: key_load_private_type: incorrect passphrase supplied to decrypt private key
          debug2: we did not send a packet, disable method
          debug1: No more authentication methods to try.
          Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
          

          and the command line used by Ansible:

          EXEC sshpass -d7 ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o 'IdentityFile="/path/to/private_key"' -o User=user -o ConnectTimeout=10 -o BatchMode=yes -o ControlPath=/some/.ansible/path remotehost '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
          
          Show
          bardelotnzl Noël Bardelot added a comment - - edited Running this from the command line fails: export SSH_ASKPASS= "/path/to/ssh_askpass.sh" export DISPLAY= ":123.456" export ANSIBLE_CONFIG=/path/to/ansible.cfg ansible-playbook /path/to/some.yml \ -i /path/to/hosts \ -- private -key /path/to/private_key \ -u user \ --ssh-extra-args= "-o BatchMode=yes" \ -vvvvv with: debug1: Offering RSA public key: /path/to/private_key debug3: send_pubkey_test debug2: we sent a publickey packet, wait for reply debug1: Server accepts key: pkalg ssh-rsa blen 279 debug2: input_userauth_pk_ok: ... debug3: sign_and_send_pubkey: RSA ... debug1: key_load_private_type: incorrect passphrase supplied to decrypt private key debug2: we did not send a packet, disable method debug1: No more authentication methods to try . Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). and the command line used by Ansible: EXEC sshpass -d7 ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o 'IdentityFile= "/path/to/private_key" ' -o User=user -o ConnectTimeout=10 -o BatchMode=yes -o ControlPath=/some/.ansible/path remotehost '/bin/sh -c ' " '" ' echo ~ && sleep 0 ' "' " ''
          Hide
          bardelotnzl Noël Bardelot added a comment - - edited

          As the command crafted by Ansible suggests, it is trying to supply the private key's passphrase using sshpass. The sshpass documentation does not explicitely mention such use case.

          If I try to run sshpass manually:

          sshpass -p thepassphrase ssh -vvv -i /path/to/private_key user@host uname
          

          I get:

          debug1: Next authentication method: publickey
          debug1: Offering RSA public key: /path/to/private_key
          debug3: send_pubkey_test
          debug2: we sent a publickey packet, wait for reply
          debug1: Server accepts key: pkalg ssh-rsa blen 279
          debug2: input_userauth_pk_ok: ...
          debug3: sign_and_send_pubkey: RSA ...
          debug1: key_load_private_type: incorrect passphrase supplied to decrypt private key
          

          and then it hangs. Thus, I think this is a bug with sshpass, or (as sshpass was not meant to be used that way) an Ansible bug :\

          Show
          bardelotnzl Noël Bardelot added a comment - - edited As the command crafted by Ansible suggests, it is trying to supply the private key's passphrase using sshpass. The sshpass documentation does not explicitely mention such use case. If I try to run sshpass manually: sshpass -p thepassphrase ssh -vvv -i /path/to/private_key user@host uname I get: debug1: Next authentication method: publickey debug1: Offering RSA public key: /path/to/private_key debug3: send_pubkey_test debug2: we sent a publickey packet, wait for reply debug1: Server accepts key: pkalg ssh-rsa blen 279 debug2: input_userauth_pk_ok: ... debug3: sign_and_send_pubkey: RSA ... debug1: key_load_private_type: incorrect passphrase supplied to decrypt private key and then it hangs. Thus, I think this is a bug with sshpass, or (as sshpass was not meant to be used that way) an Ansible bug :\

            People

            • Assignee:
              sirot Jean-Christophe Sirot
              Reporter:
              bardelotnzl Noël Bardelot
            • Votes:
              2 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated: