Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-37899

Git client does not call CredentialsProvider.snapshot() when adding credentials to a SmartCredentialsProvider that will be used on a remote instance

    Details

    • Similar Issues:

      Description

      Some properties of a credential may need to be resolved at point of use rather than being stored in the credential itself.

      Observed the following as an example stacktraces:

      With JGit as the client:

      java.lang.NullPointerException
      	at jenkins.security.ConfidentialStore.get(ConfidentialStore.java:65)
      	at jenkins.security.ConfidentialKey.load(ConfidentialKey.java:46)
      	at jenkins.security.CryptoConfidentialKey.getKey(CryptoConfidentialKey.java:32)
      	at jenkins.security.CryptoConfidentialKey.decrypt(CryptoConfidentialKey.java:67)
      	at hudson.util.Secret.decrypt(Secret.java:151)
      	at hudson.util.Secret.fromString(Secret.java:200)
      	at REDACTED.getPassword(REDACTED.java:136)
      	at org.jenkinsci.plugins.gitclient.trilead.SmartCredentialsProvider.get(SmartCredentialsProvider.java:132)
      	at org.eclipse.jgit.transport.HttpAuthMethod.authorize(HttpAuthMethod.java:219)
      	at org.eclipse.jgit.transport.TransportHttp.connect(TransportHttp.java:502)
      	at org.eclipse.jgit.transport.TransportHttp.openFetch(TransportHttp.java:309)
      	at org.eclipse.jgit.transport.FetchProcess.executeImp(FetchProcess.java:136)
      	at org.eclipse.jgit.transport.FetchProcess.execute(FetchProcess.java:122)
      	at org.eclipse.jgit.transport.Transport.fetch(Transport.java:1138)
      	at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:130)
      	at org.jenkinsci.plugins.gitclient.JGitAPIImpl$5.execute(JGitAPIImpl.java:1448)
      	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:152)
      	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:145)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:153)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:50)
      	at hudson.remoting.Request$2.run(Request.java:332)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      	at ......remote call to mac-os(Native Method)
      	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
      	at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
      	at hudson.remoting.Channel.call(Channel.java:781)
      	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:145)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:131)
      	at com.sun.proxy.$Proxy133.execute(Unknown Source)
      	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1046)
      	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1086)
      	at hudson.scm.SCM.checkout(SCM.java:495)
      	at hudson.model.AbstractProject.checkout(AbstractProject.java:1269)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
      	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
      	at hudson.model.Run.execute(Run.java:1741)
      	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      	at hudson.model.ResourceController.execute(ResourceController.java:98)
      	at hudson.model.Executor.run(Executor.java:410)
      

      with CLI Git as the client

      FATAL: null
      java.lang.NullPointerException
      	at jenkins.security.ConfidentialStore.get(ConfidentialStore.java:65)
      	at jenkins.security.ConfidentialKey.load(ConfidentialKey.java:46)
      	at jenkins.security.CryptoConfidentialKey.getKey(CryptoConfidentialKey.java:32)
      	at jenkins.security.CryptoConfidentialKey.decrypt(CryptoConfidentialKey.java:67)
      	at hudson.util.Secret.decrypt(Secret.java:151)
      	at hudson.util.Secret.fromString(Secret.java:200)
      	at REDACTED.getPassword(REDACTED.java:136)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.getGitCredentialsURL(CliGitAPIImpl.java:2635)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1383)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:63)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:314)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:506)
      	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:152)
      	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:145)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:153)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:50)
      	at hudson.remoting.Request$2.run(Request.java:332)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      	at ......remote call to mac-os(Native Method)
      	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
      	at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
      	at hudson.remoting.Channel.call(Channel.java:781)
      	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:145)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:131)
      	at com.sun.proxy.$Proxy133.execute(Unknown Source)
      	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1046)
      	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1086)
      	at hudson.scm.SCM.checkout(SCM.java:495)
      	at hudson.model.AbstractProject.checkout(AbstractProject.java:1269)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
      	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
      	at hudson.model.Run.execute(Run.java:1741)
      	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      	at hudson.model.ResourceController.execute(ResourceController.java:98)
      	at hudson.model.Executor.run(Executor.java:410)
      Finished: FAILURE
      

      In this case the REDACTED class was attempting to make a call to retrieve the password from an external password store... but that call cannot take place when working on a remote system... so in that case it returns null as the password string that then has to be encrypted into a Secret...

      Now there seems to be another issues here also... Namely a remote agent can never decrypt a Secret so that is a set-up for failure...

      So I think what should be happening is that the SmartCredentialsProvider should be Channel.exported() rather than transferred over the wire... that way it will access the credentials on the master and perform the password decryption on the one node that can decrypt a secret... then the plain text can be sent to the remote node (of course for security you want to ensure you are using either SSH agents or JNLP protocol 3/4+)... that removes the need to CredentialsProvider.snapshot() and should fix issues.

      Similarly CliGitAPIImpl.getGitCredentialsURL seems to have the same broken assumption.

      I suspect also that I have hit upon the root cause as to Password protected SSH Keys not working with the Git on remote agents... namely that the getPassword().getPlainText() is being called on the remote agent and not on the master... and hence the password cannot be decrypted and hence the ssh key cannot be used.

        Attachments

          Activity

          Hide
          markewaite Mark Waite added a comment -

          Thanks for the insight. Passphrase protected private keys in the git plugin have been a source of puzzlement to me for a very long time. There are several bug reports that they don't work in various ways. Oddly, I have a docker instance which I run in two very different environments where passphrase protected private keys work on slaves, but don't work on the master (of that docker instance). I've spent time creating various intentionally interesting ssh passphrases (trivial characters, interesting special characters, etc.), yet still haven't understood the failure cases well enough.

          I can't explain why my passphrase protected slave cases work, since I think they are the precise case you're describing should fail. I'll investigate further while evaluating git client PR207

          Show
          markewaite Mark Waite added a comment - Thanks for the insight. Passphrase protected private keys in the git plugin have been a source of puzzlement to me for a very long time. There are several bug reports that they don't work in various ways. Oddly, I have a docker instance which I run in two very different environments where passphrase protected private keys work on slaves, but don't work on the master (of that docker instance). I've spent time creating various intentionally interesting ssh passphrases (trivial characters, interesting special characters, etc.), yet still haven't understood the failure cases well enough. I can't explain why my passphrase protected slave cases work, since I think they are the precise case you're describing should fail. I'll investigate further while evaluating git client PR207
          Hide
          cleclerc Cyrille Le Clerc added a comment -
          Show
          cleclerc Cyrille Le Clerc added a comment - Pull Request submitted: https://github.com/jenkinsci/git-client-plugin/pull/235
          Hide
          markewaite Mark Waite added a comment -

          I'm still unable to make the git plugin work with a passphrase protected ssh private key on a local or a remote agent.

          Were you able to confirm that the code was not working before your change, and that it is now working after your change?

          Can you provide more details of the configuration steps you're using?

          Show
          markewaite Mark Waite added a comment - I'm still unable to make the git plugin work with a passphrase protected ssh private key on a local or a remote agent. Were you able to confirm that the code was not working before your change, and that it is now working after your change? Can you provide more details of the configuration steps you're using?
          Hide
          cleclerc Cyrille Le Clerc added a comment - - edited

          > I'm still unable to make the git plugin work with a passphrase protected ssh private key on a local or a remote agent.

          Here are my tests, is it what you mean?

          SSH Key Build on Result
           Private Key=Enter directly, passphrase=no  master  
           Private Key=Enter directly, passphrase=yes  master  
           Private Key=Enter directly, passphrase=yes  remote build agent via ssh  
           Private Key=From a file on Jenkins master, passphrase=yes  remote build agent via ssh  
           Private Key=From a file on Jenkins master, passphrase=no  remote build agent via ssh  

          Environment

          Git client plugin (git-client): 2.3.0-SNAPSHOT -> latest from github
          Ant Plugin (ant): 1.2
          OWASP Markup Formatter Plugin (antisamy-markup-formatter): 1.1
          Credentials Plugin (credentials): 2.1.12
          CVS Plug-in (cvs): 2.11
          Display URL API (display-url-api): 1.1.1
          External Monitor Job Type Plugin (external-monitor-job): 1.4
          GIT server Plugin (git-server): 1.7
          Git plugin (git): 3.0.5
          Javadoc Plugin (javadoc): 1.1
          JUnit Plugin (junit): 1.20
          LDAP Plugin (ldap): 1.11
          Mailer Plugin (mailer): 1.19
          Matrix Authorization Strategy Plugin (matrix-auth): 1.1
          Matrix Project Plugin (matrix-project): 1.8
          Maven Integration plugin (maven-plugin): 2.7.1
          PAM Authentication plugin (pam-auth): 1.1
          SCM API Plugin (scm-api): 2.0.7
          Script Security Plugin (script-security): 1.13
          SSH Credentials Plugin (ssh-credentials): 1.12
          SSH Slaves plugin (ssh-slaves): 1.9
          Structs Plugin (structs): 1.5
          Subversion Plug-in (subversion): 1.54
          Translation Assistance plugin (translation): 1.10
          Windows Slaves Plugin (windows-slaves): 1.0
          Pipeline: SCM Step (workflow-scm-step): 1.14.2
          Pipeline: Step API (workflow-step-api): 1.14.2
          
          Show
          cleclerc Cyrille Le Clerc added a comment - - edited > I'm still unable to make the git plugin work with a passphrase protected ssh private key on a local or a remote agent. Here are my tests, is it what you mean? SSH Key Build on Result  Private Key=Enter directly, passphrase=no  master    Private Key=Enter directly, passphrase=yes  master    Private Key=Enter directly, passphrase=yes  remote build agent via ssh    Private Key=From a file on Jenkins master, passphrase=yes  remote build agent via ssh    Private Key=From a file on Jenkins master, passphrase=no  remote build agent via ssh   Environment Git client plugin (git-client): 2.3.0-SNAPSHOT -> latest from github Ant Plugin (ant): 1.2 OWASP Markup Formatter Plugin (antisamy-markup-formatter): 1.1 Credentials Plugin (credentials): 2.1.12 CVS Plug-in (cvs): 2.11 Display URL API (display-url-api): 1.1.1 External Monitor Job Type Plugin (external-monitor-job): 1.4 GIT server Plugin (git-server): 1.7 Git plugin (git): 3.0.5 Javadoc Plugin (javadoc): 1.1 JUnit Plugin (junit): 1.20 LDAP Plugin (ldap): 1.11 Mailer Plugin (mailer): 1.19 Matrix Authorization Strategy Plugin (matrix-auth): 1.1 Matrix Project Plugin (matrix-project): 1.8 Maven Integration plugin (maven-plugin): 2.7.1 PAM Authentication plugin (pam-auth): 1.1 SCM API Plugin (scm-api): 2.0.7 Script Security Plugin (script-security): 1.13 SSH Credentials Plugin (ssh-credentials): 1.12 SSH Slaves plugin (ssh-slaves): 1.9 Structs Plugin (structs): 1.5 Subversion Plug-in (subversion): 1.54 Translation Assistance plugin (translation): 1.10 Windows Slaves Plugin (windows-slaves): 1.0 Pipeline: SCM Step (workflow-scm-step): 1.14.2 Pipeline: Step API (workflow-step-api): 1.14.2
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Cyrille Le Clerc
          Path:
          src/main/java/org/jenkinsci/plugins/gitclient/RemoteGitImpl.java
          http://jenkins-ci.org/commit/git-client-plugin/0381412bdd3ea7639cc6cdbb708fd03dc6a38339
          Log:
          JENKINS-37899 RemoteGitImpl: Take a snapshot of the credentials before passing them to the git client proxy

          We need to snapshot of the credentials before passing them to the git client proxy so that the credentials are snapshotted before being serialised and transferred to the build agent through Jenkins remoting.

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Cyrille Le Clerc Path: src/main/java/org/jenkinsci/plugins/gitclient/RemoteGitImpl.java http://jenkins-ci.org/commit/git-client-plugin/0381412bdd3ea7639cc6cdbb708fd03dc6a38339 Log: JENKINS-37899 RemoteGitImpl: Take a snapshot of the credentials before passing them to the git client proxy We need to snapshot of the credentials before passing them to the git client proxy so that the credentials are snapshotted before being serialised and transferred to the build agent through Jenkins remoting.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Mark Waite
          Path:
          src/main/java/org/jenkinsci/plugins/gitclient/RemoteGitImpl.java
          http://jenkins-ci.org/commit/git-client-plugin/e75b299c645b2155f693897754bb6a16501a3117
          Log:
          Merge pull request #235 from cyrille-leclerc/JENKINS-37899

          JENKINS-37899 RemoteGitImpl: Take a snapshot of the credentials before passing them to the git client proxy

          Compare: https://github.com/jenkinsci/git-client-plugin/compare/196ed5dbb394...e75b299c645b

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Mark Waite Path: src/main/java/org/jenkinsci/plugins/gitclient/RemoteGitImpl.java http://jenkins-ci.org/commit/git-client-plugin/e75b299c645b2155f693897754bb6a16501a3117 Log: Merge pull request #235 from cyrille-leclerc/ JENKINS-37899 JENKINS-37899 RemoteGitImpl: Take a snapshot of the credentials before passing them to the git client proxy Compare: https://github.com/jenkinsci/git-client-plugin/compare/196ed5dbb394...e75b299c645b
          Hide
          markewaite Mark Waite added a comment -

          Change will be included in git client plugin 2.3.0, slated to be released within the next week

          Show
          markewaite Mark Waite added a comment - Change will be included in git client plugin 2.3.0, slated to be released within the next week
          Hide
          markewaite Mark Waite added a comment -

          git client plugin 2.3.0 has released with this fix.

          Show
          markewaite Mark Waite added a comment - git client plugin 2.3.0 has released with this fix.

            People

            • Assignee:
              markewaite Mark Waite
              Reporter:
              stephenconnolly Stephen Connolly
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: