Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-32834

Failed to connect to repository with password-protected ssh key

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Blocker
    • Resolution: Fixed
    • Component/s: git-client-plugin
    • Environment:
    • Similar Issues:

      Description

      Cannot connect to repository after upgrading to git-client-plugin 1.19.3. Downgrading to 1.19.2 fixes the issue. The key used to connect to the repository is password-protected.

      Logging-in into Jenkins and executing following command succeeds:

      ssh-agent bash -c 'ssh-add ~/.ssh/jenkins.opencms-core.id_rsa; git -c core.askpass=true ls-remote ssh://git@mygithubenterprise:myrepo/tests.git'

      GUI Error message:

      Failed to connect to repository : Command "git -c core.askpass=true ls-remote -h git@mygithubenterprise:myrepo/tests.git HEAD" returned status code 128:
      stdout: 
      stderr: Permission denied (publickey). 
      fatal: Could not read from remote repository.
      
      Please make sure you have the correct access rights
      and the repository exists.
      
      

      Console output if forcing build:

      Started by user myuser
      Building in workspace /srv/jenkins/jobs/Tests/jobs/kk/workspace
      Wiping out workspace first.
      Cloning the remote Git repository
      Cloning repository git@mygithubenterprise:myrepo/tests.git
       > git init /srv/jenkins/jobs/Tests/jobs/kk/workspace # timeout=10
      Fetching upstream changes from git@mygithubenterprise:myrepo/tests.git
       > git --version # timeout=10
      using GIT_SSH to set credentials jenkins.tests.id_rsa pass
       > git -c core.askpass=true fetch --tags --progress git@mygithubenterprise:myrepo/tests.git +refs/heads/*:refs/remotes/origin/*
      ERROR: Error cloning remote repo 'origin'
      hudson.plugins.git.GitException: Command "git -c core.askpass=true fetch --tags --progress git@mygithubenterprise:myrepo/tests.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
      stdout: 
      stderr: Permission denied (publickey).
      fatal: Could not read from remote repository.
      
      Please make sure you have the correct access rights
      and the repository exists.
      
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1710)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1454)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:63)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:314)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:506)
      	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1013)
      	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1053)
      	at hudson.scm.SCM.checkout(SCM.java:485)
      	at hudson.model.AbstractProject.checkout(AbstractProject.java:1269)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
      	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
      	at hudson.model.Run.execute(Run.java:1738)
      	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      	at hudson.model.ResourceController.execute(ResourceController.java:98)
      	at hudson.model.Executor.run(Executor.java:410)
      

        Attachments

          Activity

          Hide
          markewaite Mark Waite added a comment -

          So sorry that it regressed from 1.19.2 to 1.19.3. I can't duplicate the condition to make the plugin access a repository using a passphrase protected private key, even after confirming that the passphrase protected private key works with command line git.

          I'd like to confirm the problem is due to a single commit, and I'd like to understand more about your configuration to see how you've made passphrase protected private keys work with Jenkins.

          Would you be willing to install and test a 1.19.4 snapshot build which reverts that commit?

          Show
          markewaite Mark Waite added a comment - So sorry that it regressed from 1.19.2 to 1.19.3. I can't duplicate the condition to make the plugin access a repository using a passphrase protected private key, even after confirming that the passphrase protected private key works with command line git. I'd like to confirm the problem is due to a single commit , and I'd like to understand more about your configuration to see how you've made passphrase protected private keys work with Jenkins. Would you be willing to install and test a 1.19.4 snapshot build which reverts that commit?
          Hide
          agallardo Alberto Gallardo added a comment -

          Of course.

          Show
          agallardo Alberto Gallardo added a comment - Of course.
          Hide
          agallardo Alberto Gallardo added a comment -

          Thanks! I have tested again git-client-1.19.3 reverting your mentioned commit and I can confirm that it works again.

          Show
          agallardo Alberto Gallardo added a comment - Thanks! I have tested again git-client-1.19.3 reverting your mentioned commit and I can confirm that it works again.
          Hide
          markewaite Mark Waite added a comment -

          Thanks. Short term, I will release a new version of the plugin with that change removed.

          I'd really like to understand how your system is configured. I haven't yet found a consistent way to use a passphrase protected private key with a Jenkins job. Would you be willing to either upload the job definition (config.xml file), or describe the settings used in the job definition?

          Some of the questions that come to mind for me include:

          • Are you using the ssh-agent plugin?
          • Are you running ssh-agent separately from the Jenkins master or slave?
          • Are you using passphrase based private keys on slaves or only on the master?
          • Which operating systems are you using when the passphrase base private keys are successful?
          • Are there other hints you can offer that would explain how it works for you, so that I can duplicate that (and include it in an automated test)?
          Show
          markewaite Mark Waite added a comment - Thanks. Short term, I will release a new version of the plugin with that change removed. I'd really like to understand how your system is configured. I haven't yet found a consistent way to use a passphrase protected private key with a Jenkins job. Would you be willing to either upload the job definition (config.xml file), or describe the settings used in the job definition? Some of the questions that come to mind for me include: Are you using the ssh-agent plugin? Are you running ssh-agent separately from the Jenkins master or slave? Are you using passphrase based private keys on slaves or only on the master? Which operating systems are you using when the passphrase base private keys are successful? Are there other hints you can offer that would explain how it works for you, so that I can duplicate that (and include it in an automated test)?
          Hide
          agallardo Alberto Gallardo added a comment - - edited

          Example jenkins config - JENKINS#32834: config.xml

          Show
          agallardo Alberto Gallardo added a comment - - edited Example jenkins config - JENKINS#32834: config.xml
          Hide
          agallardo Alberto Gallardo added a comment -
          • Are you using the ssh-agent plugin?
            yes: ssh agent plugin 1.9
          • Are you running ssh-agent separately from the Jenkins master or slave?
            At the moment, I only have a master
          • Are you using passphrase based private keys on slaves or only on the master?
            Only in master
          • Which operating systems are you using when the passphrase base private keys are successful?
            Linux 3.0.101 64bit (SLES)
          • Are there other hints you can offer that would explain how it works for you, so that I can duplicate that (and include it in an automated test)?
            hummm... I can reproduce it always. I can try to play a bit with the ssh plugins. These are the ssh plugins I have:
            SSH Agent Plugin 1.9
            SSH Credentials Plugin 1.11
            SSH Slaves plugin 1.10
          Show
          agallardo Alberto Gallardo added a comment - Are you using the ssh-agent plugin? yes: ssh agent plugin 1.9 Are you running ssh-agent separately from the Jenkins master or slave? At the moment, I only have a master Are you using passphrase based private keys on slaves or only on the master? Only in master Which operating systems are you using when the passphrase base private keys are successful? Linux 3.0.101 64bit (SLES) Are there other hints you can offer that would explain how it works for you, so that I can duplicate that (and include it in an automated test)? hummm... I can reproduce it always. I can try to play a bit with the ssh plugins. These are the ssh plugins I have: SSH Agent Plugin 1.9 SSH Credentials Plugin 1.11 SSH Slaves plugin 1.10
          Hide
          agallardo Alberto Gallardo added a comment -

          Notice that I'm not using the "Build Environment > SSH Agent" option. I have always assumed that this is for build triggered ssh commands, and not for logging into the git(hub) repository.

          Show
          agallardo Alberto Gallardo added a comment - Notice that I'm not using the "Build Environment > SSH Agent" option. I have always assumed that this is for build triggered ssh commands, and not for logging into the git(hub) repository.
          Hide
          agallardo Alberto Gallardo added a comment -

          More environment info: I'm using the jenkins on the embedded winstone server.

          $zypper info jenkins
          [...]
          Name: jenkins
          Version: 1.647-1.2
          Arch: noarch
          [...]
          
          $java -version
          java version "1.7.0_79"
          Java(TM) SE Runtime Environment (build 1.7.0_79-b15)
          Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)
          
          # winstone.jar/META-INF/MANIFEST.MF:
          [...]
          Implementation-Title: Winstone
          Implementation-Version: 2.9
          [...]
          
          Show
          agallardo Alberto Gallardo added a comment - More environment info: I'm using the jenkins on the embedded winstone server. $zypper info jenkins [...] Name: jenkins Version: 1.647-1.2 Arch: noarch [...] $java -version java version "1.7.0_79" Java(TM) SE Runtime Environment (build 1.7.0_79-b15) Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode) # winstone.jar/META-INF/MANIFEST.MF: [...] Implementation-Title: Winstone Implementation-Version: 2.9 [...]
          Hide
          agallardo Alberto Gallardo added a comment -

          Apparently, there is not much more I can test: I have disabled the 'SSH Slaves plugin' (I don't have slaves). The GUI renders the error independently of the 'Build Environment > SSH Agent' setting. If it helps, when setting this 'SSH Agent' option, the job fails with this message (I have to launch it manually):

          Started by user ...
          Building in workspace /srv/jenkins/jobs/Tests/jobs/test_github_jenkins_integration_ghp/workspace
          [ssh-agent] Looking for ssh-agent implementation...
          [ssh-agent]   Java/JNR ssh-agent
          [ssh-agent] Started.
          [ssh-agent] Unable to read key: Unable to create OpenSSL PBDKF: PBKDF-OpenSSL SecretKeyFactory not available
          org.bouncycastle.openssl.PEMException: Unable to create OpenSSL PBDKF: PBKDF-OpenSSL SecretKeyFactory not available
          	at org.bouncycastle.openssl.jcajce.PEMUtilities.getKey(Unknown Source)
          	at org.bouncycastle.openssl.jcajce.PEMUtilities.getKey(Unknown Source)
          	at org.bouncycastle.openssl.jcajce.PEMUtilities.crypt(Unknown Source)
          	at org.bouncycastle.openssl.jcajce.JcePEMDecryptorProviderBuilder$1$1.decrypt(Unknown Source)
          	at org.bouncycastle.openssl.PEMEncryptedKeyPair.decryptKeyPair(Unknown Source)
          	at com.cloudbees.jenkins.plugins.sshagent.jna.JNRRemoteAgent.addIdentity(JNRRemoteAgent.java:98)
          	at com.cloudbees.jenkins.plugins.sshagent.SSHAgentBuildWrapper$SSHAgentEnvironment.add(SSHAgentBuildWrapper.java:388)
          	at com.cloudbees.jenkins.plugins.sshagent.SSHAgentBuildWrapper.preCheckout(SSHAgentBuildWrapper.java:227)
          	at jenkins.scm.SCMCheckoutStrategy.preCheckout(SCMCheckoutStrategy.java:76)
          	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:528)
          	at hudson.model.Run.execute(Run.java:1738)
          	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          	at hudson.model.ResourceController.execute(ResourceController.java:98)
          	at hudson.model.Executor.run(Executor.java:410)
          Caused by: java.security.NoSuchAlgorithmException: PBKDF-OpenSSL SecretKeyFactory not available
          	at javax.crypto.SecretKeyFactory.<init>(SecretKeyFactory.java:121)
          	at javax.crypto.SecretKeyFactory.getInstance(SecretKeyFactory.java:159)
          	at org.bouncycastle.jcajce.util.DefaultJcaJceHelper.createSecretKeyFactory(Unknown Source)
          	... 14 more
          [ssh-agent] Using credentials jenkins (jenkins.tests.id_rsa pass)
          Wiping out workspace first.
          Cloning the remote Git repository
          Cloning repository git@github.tik.uni-stuttgart.de:NFL/tests.git
           > git init /srv/jenkins/jobs/Tests/jobs/test_github_jenkins_integration_ghp/workspace # timeout=10
          Fetching upstream changes from git@github.tik.uni-stuttgart.de:NFL/tests.git
           > git --version # timeout=10
          using GIT_SSH to set credentials jenkins.tests.id_rsa pass
           > git -c core.askpass=true fetch --tags --progress git@github.tik.uni-stuttgart.de:NFL/tests.git +refs/heads/*:refs/remotes/origin/*
          ERROR: Error cloning remote repo 'origin'
          hudson.plugins.git.GitException: Command "git -c core.askpass=true fetch --tags --progress git@github.tik.uni-stuttgart.de:NFL/tests.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
          stdout: 
          stderr: Permission denied (publickey).
          fatal: Could not read from remote repository.
          
          Please make sure you have the correct access rights
          and the repository exists.
          
          Show
          agallardo Alberto Gallardo added a comment - Apparently, there is not much more I can test: I have disabled the 'SSH Slaves plugin' (I don't have slaves). The GUI renders the error independently of the 'Build Environment > SSH Agent' setting. If it helps, when setting this 'SSH Agent' option, the job fails with this message (I have to launch it manually): Started by user ... Building in workspace /srv/jenkins/jobs/Tests/jobs/test_github_jenkins_integration_ghp/workspace [ssh-agent] Looking for ssh-agent implementation... [ssh-agent] Java/JNR ssh-agent [ssh-agent] Started. [ssh-agent] Unable to read key: Unable to create OpenSSL PBDKF: PBKDF-OpenSSL SecretKeyFactory not available org.bouncycastle.openssl.PEMException: Unable to create OpenSSL PBDKF: PBKDF-OpenSSL SecretKeyFactory not available at org.bouncycastle.openssl.jcajce.PEMUtilities.getKey(Unknown Source) at org.bouncycastle.openssl.jcajce.PEMUtilities.getKey(Unknown Source) at org.bouncycastle.openssl.jcajce.PEMUtilities.crypt(Unknown Source) at org.bouncycastle.openssl.jcajce.JcePEMDecryptorProviderBuilder$1$1.decrypt(Unknown Source) at org.bouncycastle.openssl.PEMEncryptedKeyPair.decryptKeyPair(Unknown Source) at com.cloudbees.jenkins.plugins.sshagent.jna.JNRRemoteAgent.addIdentity(JNRRemoteAgent.java:98) at com.cloudbees.jenkins.plugins.sshagent.SSHAgentBuildWrapper$SSHAgentEnvironment.add(SSHAgentBuildWrapper.java:388) at com.cloudbees.jenkins.plugins.sshagent.SSHAgentBuildWrapper.preCheckout(SSHAgentBuildWrapper.java:227) at jenkins.scm.SCMCheckoutStrategy.preCheckout(SCMCheckoutStrategy.java:76) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:528) at hudson.model.Run.execute(Run.java:1738) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:410) Caused by: java.security.NoSuchAlgorithmException: PBKDF-OpenSSL SecretKeyFactory not available at javax.crypto.SecretKeyFactory.<init>(SecretKeyFactory.java:121) at javax.crypto.SecretKeyFactory.getInstance(SecretKeyFactory.java:159) at org.bouncycastle.jcajce.util.DefaultJcaJceHelper.createSecretKeyFactory(Unknown Source) ... 14 more [ssh-agent] Using credentials jenkins (jenkins.tests.id_rsa pass) Wiping out workspace first. Cloning the remote Git repository Cloning repository git@github.tik.uni-stuttgart.de:NFL/tests.git > git init /srv/jenkins/jobs/Tests/jobs/test_github_jenkins_integration_ghp/workspace # timeout=10 Fetching upstream changes from git@github.tik.uni-stuttgart.de:NFL/tests.git > git --version # timeout=10 using GIT_SSH to set credentials jenkins.tests.id_rsa pass > git -c core.askpass=true fetch --tags --progress git@github.tik.uni-stuttgart.de:NFL/tests.git +refs/heads/*:refs/remotes/origin/* ERROR: Error cloning remote repo 'origin' hudson.plugins.git.GitException: Command "git -c core.askpass=true fetch --tags --progress git@github.tik.uni-stuttgart.de:NFL/tests.git +refs/heads/*:refs/remotes/origin/*" returned status code 128: stdout: stderr: Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Mark Waite
          Path:
          README.md
          http://jenkins-ci.org/commit/git-client-plugin/fda981f35a28683f53b480ffad779850bfaebc39
          Log:
          [Fix JENKINS-32834] don't break ssh passphrase in few working cases

          Refer to 7711aacb34aa597a92aad14e8d7429347a920b8d

          README change because I made a mistake in commit message for 7711aacb

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Mark Waite Path: README.md http://jenkins-ci.org/commit/git-client-plugin/fda981f35a28683f53b480ffad779850bfaebc39 Log: [Fix JENKINS-32834] don't break ssh passphrase in few working cases Refer to 7711aacb34aa597a92aad14e8d7429347a920b8d README change because I made a mistake in commit message for 7711aacb
          Hide
          markewaite Mark Waite added a comment - - edited

          I'm still perplexed how you make it work in your environment.

          I added the ssh-agent plugin to my Jenkins installation, modified the job I had previously defined to use a passphrase protected credential for its git repository, then selected that same passphrase protected credential for use by the ssh-agent in the "Build Environment" section of the job configuration. I then encountered what appears to be JENKINS-32120 (ssh-agent fails when using a passphrase protected key).

          I'm even more perplexed because your config.xml does not include the ssh-agent build wrapper that I was expecting. Are you sure that is the job configuration file of one of the jobs using ssh-agent?

          Show
          markewaite Mark Waite added a comment - - edited I'm still perplexed how you make it work in your environment. I added the ssh-agent plugin to my Jenkins installation, modified the job I had previously defined to use a passphrase protected credential for its git repository, then selected that same passphrase protected credential for use by the ssh-agent in the " Build Environment " section of the job configuration. I then encountered what appears to be JENKINS-32120 (ssh-agent fails when using a passphrase protected key). I'm even more perplexed because your config.xml does not include the ssh-agent build wrapper that I was expecting. Are you sure that is the job configuration file of one of the jobs using ssh-agent?
          Hide
          markewaite Mark Waite added a comment -

          Fixed in git client plugin 1.19.4, released 9 Feb 2016

          Show
          markewaite Mark Waite added a comment - Fixed in git client plugin 1.19.4, released 9 Feb 2016
          Hide
          agallardo Alberto Gallardo added a comment -

          I'm still perplexed how you make it work in your environment.

          Re-reading my posts, I'm afraid I have confused you a bit. I'm sorry. "I'm using the ssh-agent" as in "I have installed it" (it's a GitHub Pull Request Builder dependency). The problem is that I'm not that familiarized with each plugin responsibility, and I assumed that it was a jenkins core component already being used.

          I added the ssh-agent plugin to my Jenkins installation, modified the job I had previously defined to use a passphrase protected credential for its git repository, then selected that same passphrase protected credential for use by the ssh-agent in the "Build Environment" section of the job configuration. I then encountered what appears to be JENKINS-32120 (ssh-agent fails when using a passphrase protected key).

          That confirms my findings in my previous post. Thanks!

          I'm even more perplexed because your config.xml does not include the ssh-agent build wrapper that I was expecting. Are you sure that is the job configuration file of one of the jobs using ssh-agent?

          My fault making you believe I was using it: no, I'm not using it in this job.

          I have now re-tested, removing the "ssh agent" plugin (and consequently, the "github pull request builder" plugin). The problem is always reproducible, and I can confirm that the commit revert still works (built git-client git-client-1.19.4 from github). Unfortunatelly, my jenkins doesn't see the new 1.19.4 release yet.

          Show
          agallardo Alberto Gallardo added a comment - I'm still perplexed how you make it work in your environment. Re-reading my posts, I'm afraid I have confused you a bit. I'm sorry. " I'm using the ssh-agent " as in " I have installed it " (it's a GitHub Pull Request Builder dependency). The problem is that I'm not that familiarized with each plugin responsibility, and I assumed that it was a jenkins core component already being used. I added the ssh-agent plugin to my Jenkins installation, modified the job I had previously defined to use a passphrase protected credential for its git repository, then selected that same passphrase protected credential for use by the ssh-agent in the "Build Environment" section of the job configuration. I then encountered what appears to be JENKINS-32120 (ssh-agent fails when using a passphrase protected key). That confirms my findings in my previous post . Thanks! I'm even more perplexed because your config.xml does not include the ssh-agent build wrapper that I was expecting. Are you sure that is the job configuration file of one of the jobs using ssh-agent? My fault making you believe I was using it: no, I'm not using it in this job. I have now re-tested, removing the "ssh agent" plugin (and consequently, the "github pull request builder" plugin). The problem is always reproducible, and I can confirm that the commit revert still works (built git-client git-client-1.19.4 from github). Unfortunatelly, my jenkins doesn't see the new 1.19.4 release yet.
          Hide
          agallardo Alberto Gallardo added a comment -

          If it helps, I have the 'SSH Credentials Plugin' v. 1.11 installed.

          Show
          agallardo Alberto Gallardo added a comment - If it helps, I have the 'SSH Credentials Plugin' v. 1.11 installed.
          Hide
          vanniktech Niklas Baudy added a comment -

          I'm also having the same problems when running 1.19.3. I have Jenkins on Mac also with SSH Credentials Plugin v1.11.

          Also I can't see the 1.19.4 release yet, do you know when we'll be able to see it?

          Show
          vanniktech Niklas Baudy added a comment - I'm also having the same problems when running 1.19.3. I have Jenkins on Mac also with SSH Credentials Plugin v1.11. Also I can't see the 1.19.4 release yet, do you know when we'll be able to see it?
          Hide
          markewaite Mark Waite added a comment -

          I've confirmed the update center now shows git client plugin 1.19.4. Be sure you refresh the update center by pressing the "Check Now" button in the Jenkins Update Center web page.

          Show
          markewaite Mark Waite added a comment - I've confirmed the update center now shows git client plugin 1.19.4. Be sure you refresh the update center by pressing the "Check Now" button in the Jenkins Update Center web page.
          Hide
          agallardo Alberto Gallardo added a comment -

          Confirmed that the version from the update center works for me.

          Show
          agallardo Alberto Gallardo added a comment - Confirmed that the version from the update center works for me.

            People

            • Assignee:
              markewaite Mark Waite
              Reporter:
              agallardo Alberto Gallardo
            • Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: