Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48913

sshagent block serialization error when script with git clone fails

    Details

    • Similar Issues:

      Description

      We have a pipeline that looks something like:

      node {
          sshagent (credentials: ['ssh-id-rsa']) {
              sh 'run-somescript.sh > build.log'
           }    
      }

      Where the script checks out a git repo via ssh.

      git clone --recursive git@github.com:...

      The repo also contains a https submodule.

      Sometimes, we see this error from the submodule.

      Submodule 'module' (https://github.com/<org>/<repo>.git) registered for path 'module'
      Cloning into 'module'...
      error: index-pack died of signal 15
      fatal: index-pack failed
      fatal: clone of 'https://github.com/<org>/<repo>.git' into submodule path 'module' failed

      Somehow though, this is causing a serialization exception in the pipeline.

      an exception which occurred:
      	in field com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey.privateKeySource
      	in object com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey@ab46eb8d
      	in field com.cloudbees.groovy.cps.impl.BlockScopeEnv.locals
      	in object com.cloudbees.groovy.cps.impl.BlockScopeEnv@61a1bba7
      	in field com.cloudbees.groovy.cps.impl.CallEnv.caller
      	in object com.cloudbees.groovy.cps.impl.FunctionCallEnv@652b66c8
      	in field com.cloudbees.groovy.cps.impl.ProxyEnv.parent
      	in object com.cloudbees.groovy.cps.impl.BlockScopeEnv@6b74a32c
      	in field com.cloudbees.groovy.cps.impl.CallEnv.caller
      	in object com.cloudbees.groovy.cps.impl.FunctionCallEnv@42d99103
      	in field com.cloudbees.groovy.cps.Continuable.e
      	in object com.cloudbees.groovy.cps.Continuable@647c2b05
      	in field org.jenkinsci.plugins.workflow.cps.CpsThread.program
      	in object org.jenkinsci.plugins.workflow.cps.CpsThread@155c484c
      	in field org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.threads
      	in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@5fd58ed3
      	in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@5fd58ed3
      Also:   hudson.AbortException: script returned exit code 1
      		at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.check(DurableTaskStep.java:335)
      		at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.run(DurableTaskStep.java:289)
      		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
      		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      Caused: java.io.NotSerializableException: com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$FileOnMasterPrivateKeySource
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:860)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
      	at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
      	at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
      	at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50)
      	at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179)
      	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
      	at java.util.HashMap.internalWriteEntries(HashMap.java:1790)
      	at java.util.HashMap.writeObject(HashMap.java:1363)
      	at sun.reflect.GeneratedMethodAccessor375.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:273)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:976)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
      	at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
      	at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
      	at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50)
      	at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179)
      	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
      	at java.util.TreeMap.writeObject(TreeMap.java:2438)
      	at sun.reflect.GeneratedMethodAccessor433.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:273)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:976)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
      	at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)
      	at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)
      	at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverWriter.writeObject(RiverWriter.java:140)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:459)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:435)
      	at org.jenkinsci.plugins.workflow.cps.CpsStepContext$3.onSuccess(CpsStepContext.java:529)
      	at org.jenkinsci.plugins.workflow.cps.CpsStepContext$3.onSuccess(CpsStepContext.java:525)
      	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4$1.run(CpsFlowExecution.java:746)
      	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      
      

      Most interestingly,

      Caused: java.io.NotSerializableException: com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$FileOnMasterPrivateKeySource

        Attachments

          Issue Links

            Activity

            Hide
            jglick Jesse Glick added a comment -

            I was unable to reproduce any error like this. Apparently an BasicSSHUserPrivateKey object is somehow getting saved into the Pipeline program state, which should not happen—SSHAgentStepExecution merely loads private keys in a Java local variable, uses them to add identities to an agent, binds an SSH_AUTH_SOCK, and then runs its body. I see no plausible code path by which this type from the ssh-credentials could be “leaking” into an attempted save to program.dat. Is there any known way to reproduce from scratch, including particular versions of Jenkins core, this ssh-agent plugin, and Pipeline-related plugins such as workflow-job and workflow-cps?

            Show
            jglick Jesse Glick added a comment - I was unable to reproduce any error like this. Apparently an BasicSSHUserPrivateKey object is somehow getting saved into the Pipeline program state, which should not happen— SSHAgentStepExecution merely loads private keys in a Java local variable, uses them to add identities to an agent, binds an SSH_AUTH_SOCK , and then runs its body. I see no plausible code path by which this type from the ssh-credentials could be “leaking” into an attempted save to program.dat . Is there any known way to reproduce from scratch, including particular versions of Jenkins core, this ssh-agent plugin, and Pipeline-related plugins such as workflow-job and workflow-cps ?
            Hide
            jglick Jesse Glick added a comment -

            Note that BasicSSHUserPrivateKey.FileOnMasterPrivateKeySource is not Serializable, nor is UsersPrivateKeySource, only DirectEntryPrivateKeySource, since CredentialsSnapshotTakerImpl converts the former two into the latter. For purposes of XStream serialization, Serializable is irrelevant. For purposes of Remoting, plugin code is expected to call CredentialsProvider.snapshot before transmitting credentials to an agent. But a Credentials object is never expected to be serialized inside a Pipeline program.

            Show
            jglick Jesse Glick added a comment - Note that BasicSSHUserPrivateKey.FileOnMasterPrivateKeySource is not Serializable , nor is UsersPrivateKeySource , only DirectEntryPrivateKeySource , since CredentialsSnapshotTakerImpl converts the former two into the latter. For purposes of XStream serialization, Serializable is irrelevant. For purposes of Remoting, plugin code is expected to call CredentialsProvider.snapshot before transmitting credentials to an agent. But a Credentials object is never expected to be serialized inside a Pipeline program.
            Hide
            mr_dfuse Nico Mommaerts added a comment -

            I have just encountered the same exception using a jnlp slave. Both master and slave are docker images that worked fine before, only thing I changed was to bump the lts of the master to the latest (from 2.107.3 to 2.121.1), and upgraded some plugins (including credentials). I was still using remoting version 3.19 on the slave, I bumped that to 3.20 after I saw this exception. Exception still occured though. Then I upgraded Pipeline:Groovy and from 2.53 to 2.54, and Pipeline:Job from 2.21 to 2.22, after which the master needed to be restarted. After that the exception was gone. So the root cause would have been either in the remoting slave that needed a master restart first, or in upgrading those two plugins.

            Show
            mr_dfuse Nico Mommaerts added a comment - I have just encountered the same exception using a jnlp slave. Both master and slave are docker images that worked fine before, only thing I changed was to bump the lts of the master to the latest (from 2.107.3 to 2.121.1), and upgraded some plugins (including credentials). I was still using remoting version 3.19 on the slave, I bumped that to 3.20 after I saw this exception. Exception still occured though. Then I upgraded Pipeline:Groovy and from 2.53 to 2.54, and Pipeline:Job from 2.21 to 2.22, after which the master needed to be restarted. After that the exception was gone. So the root cause would have been either in the remoting slave that needed a master restart first, or in upgrading those two plugins.
            Hide
            croche Christian Roche added a comment -

            I seem to have bumped into a similar error with this piece of code that was working until recently

                 steps {
                      deleteDir()
                      git url: tf_url, branch: tf_branch, credentialsId: tf_creds
                  }
            

            and suddenly gave me

            java.io.NotSerializableException: com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$FileOnMasterPrivateKeySource
            

             
            The reason was plugin ssh-credentials being upgraded to version 1.14 and fixing CVE-2018-1000601:

            SSH Credentials Plugin no longer supports SSH credentials from files on the Jenkins master file system, neither user-specified file paths nor ~/.ssh. Existing SSH credentials of these kinds are migrated to "directly entered" SSH credentials.

            To fix the error I had to change my init.groovy.d code from

                 // Register an SSH private key file
                 key_creds = new BasicSSHUserPrivateKey(
                   /* key_scope       */ CredentialsScope.GLOBAL,
                   /* key_id          */ file_name,
                   /* key_username    */ "git",
                   /* key_source      */ new BasicSSHUserPrivateKey.FileOnMasterPrivateKeySource(key_file.absolutePath),
            

            to

                key_text = new File(key_file.absolutePath).text
            
                 // Register an SSH private key file
                 key_creds = new BasicSSHUserPrivateKey(
                   /* key_scope       */ CredentialsScope.GLOBAL,
                   /* key_id          */ file_name,
                   /* key_username    */ "git",
                   /* key_source      */ new BasicSSHUserPrivateKey.DirectEntryPrivateKeySource(key_text),
             
            Show
            croche Christian Roche added a comment - I seem to have bumped into a similar error with this piece of code that was working until recently steps { deleteDir() git url: tf_url, branch: tf_branch, credentialsId: tf_creds } and suddenly gave me java.io.NotSerializableException: com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$FileOnMasterPrivateKeySource   The reason was plugin ssh-credentials being upgraded to version 1.14 and fixing CVE-2018-1000601 : SSH Credentials Plugin no longer supports SSH credentials from files on the Jenkins master file system, neither user-specified file paths nor ~/.ssh. Existing SSH credentials of these kinds are migrated to "directly entered" SSH credentials. To fix the error I had to change my init.groovy.d code from // Register an SSH private key file key_creds = new BasicSSHUserPrivateKey( /* key_scope */ CredentialsScope.GLOBAL, /* key_id */ file_name, /* key_username */ "git" , /* key_source */ new BasicSSHUserPrivateKey.FileOnMasterPrivateKeySource(key_file.absolutePath), to key_text = new File(key_file.absolutePath).text // Register an SSH private key file key_creds = new BasicSSHUserPrivateKey( /* key_scope */ CredentialsScope.GLOBAL, /* key_id */ file_name, /* key_username */ "git" , /* key_source */ new BasicSSHUserPrivateKey.DirectEntryPrivateKeySource(key_text),  
            Hide
            mr_dfuse Nico Mommaerts added a comment -

            Christian Roche thanks, that fixed it for me as well. So this is probably not a bug but caused by using code that should be marked deprecated in the plugin.

            Show
            mr_dfuse Nico Mommaerts added a comment - Christian Roche thanks, that fixed it for me as well. So this is probably not a bug but caused by using code that should be marked deprecated in the plugin.

              People

              • Assignee:
                Unassigned
                Reporter:
                externl Joe George
              • Votes:
                1 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated: