Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-51191

hudson.remoting.Channel$CallSiteStackTrace: Remote call to

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Cannot Reproduce
    • Component/s: git-client-plugin, remoting
    • Labels:
      None
    • Environment:
      Jenkins 2.114
      Java 1.8.0_131
      Remoting 3.19
      git-client-plugin 2.7.1
    • Similar Issues:

      Description

      My jobs fails randomly (about 10% are failed), they are defined as pipelines. Master and jenkins slaves have exactly the same environment.

       
      Generally, build fails when it is going to load/execute pipeline script in remote machine, it ends with following message:
       
      Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to 192.168.3.27
              at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
              at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
              at hudson.remoting.Channel.call(Channel.java:955)
              at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
              at com.sun.proxy.$Proxy98.withRepository(Unknown Source)
              at org.jenkinsci.plugins.gitclient.RemoteGitImpl.withRepository(RemoteGitImpl.java:235)
              at hudson.plugins.git.GitSCM.printCommitMessageToLog(GitSCM.java:1245)
              at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1218)
              at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113)
              at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85)
              at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75)
              at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
              at hudson.security.ACL.impersonate(ACL.java:290)
              at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      java.lang.NoClassDefFoundError: hudson.model.Computer
          at org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.withRepository(AbstractGitAPIImpl.java:29)
          at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.withRepository(CliGitAPIImpl.java:72)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:498)
          at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929)
          at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903)
          at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855)
          at hudson.remoting.UserRequest.perform(UserRequest.java:212)
          at hudson.remoting.UserRequest.perform(UserRequest.java:54)
          at hudson.remoting.Request$2.run(Request.java:369)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      Caused: java.io.IOException: Remote call on 192.168.3.27 failed
          at hudson.remoting.Channel.call(Channel.java:961)
          at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
          at com.sun.proxy.$Proxy98.withRepository(Unknown Source)
          at org.jenkinsci.plugins.gitclient.RemoteGitImpl.withRepository(RemoteGitImpl.java:235)
          at hudson.plugins.git.GitSCM.printCommitMessageToLog(GitSCM.java:1245)
          at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1218)
          at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113)
          at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85)
          at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75)
          at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
          at hudson.security.ACL.impersonate(ACL.java:290)
          at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:748)
      Finished: FAILURE
       
       
      On Jenkins node log I can find:
       
       
      <===[JENKINS REMOTING CAPACITY]===>channel started
      Remoting version: 3.19
      This is a Unix agent
      Evacuated stdout
      May 04, 2018 12:23:05 AM org.jenkinsci.remoting.util.AnonymousClassWarnings warn
      WARNING: Attempt to (de-)serialize anonymous class net.bull.javamelody.RemoteCallHelper$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/
      May 04, 2018 12:23:27 AM org.jenkinsci.remoting.util.AnonymousClassWarnings warn
      WARNING: Attempt to (de-)serialize anonymous class org.jenkinsci.plugins.envinject.EnvInjectComputerListener$2; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/
      Agent successfully connected and online
      May 04, 2018 9:16:41 AM jenkins.model.Jenkins <clinit>
      SEVERE: Failed to load Jenkins.class
      com.thoughtworks.xstream.InitializationException: Could not instantiate converter : com.thoughtworks.xstream.converters.extended.DurationConverter : null
          at com.thoughtworks.xstream.XStream.registerConverterDynamically(XStream.java:904)
          at com.thoughtworks.xstream.XStream.setupConverters(XStream.java:867)
          at com.thoughtworks.xstream.XStream.<init>(XStream.java:574)
          at com.thoughtworks.xstream.XStream.<init>(XStream.java:496)
          at com.thoughtworks.xstream.XStream.<init>(XStream.java:465)
          at com.thoughtworks.xstream.XStream.<init>(XStream.java:411)
          at com.thoughtworks.xstream.XStream.<init>(XStream.java:378)
          at hudson.util.XStream2.<init>(XStream2.java:112)
          at jenkins.model.Jenkins.<clinit>(Jenkins.java:5103)
          at hudson.model.Computer.<clinit>(Computer.java:1761)
          at hudson.util.ProcessTree.getKillers(ProcessTree.java:157)
          at hudson.util.ProcessTree$OSProcess.killByKiller(ProcessTree.java:220)
          at hudson.util.ProcessTree$UnixProcess.kill(ProcessTree.java:662)
          at hudson.util.ProcessTree$UnixProcess.killRecursively(ProcessTree.java:670)
          at hudson.util.ProcessTree$UnixProcess.killRecursively(ProcessTree.java:669)
          at hudson.util.ProcessTree$UnixProcess.killRecursively(ProcessTree.java:669)
          at hudson.util.ProcessTree$Unix.killAll(ProcessTree.java:591)
          at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1090)
          at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1081)
          at hudson.remoting.UserRequest.perform(UserRequest.java:212)
          at hudson.remoting.UserRequest.perform(UserRequest.java:54)
          at hudson.remoting.Request$2.run(Request.java:369)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.reflect.InvocationTargetException
          at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
          at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
          at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
          at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
          at com.thoughtworks.xstream.XStream.registerConverterDynamically(XStream.java:897)
          ... 26 more
      Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException
          at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:299)
          at com.sun.proxy.$Proxy5.getResources2(Unknown Source)
          at hudson.remoting.RemoteClassLoader.findResources(RemoteClassLoader.java:481)
          at java.lang.ClassLoader.getResources(ClassLoader.java:1142)
          at java.util.ServiceLoader$LazyIterator.hasNextService(ServiceLoader.java:348)
          at java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:393)
          at java.util.ServiceLoader$1.hasNext(ServiceLoader.java:474)
          at javax.xml.datatype.FactoryFinder$1.run(FactoryFinder.java:296)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.xml.datatype.FactoryFinder.findServiceProvider(FactoryFinder.java:292)
          at javax.xml.datatype.FactoryFinder.find(FactoryFinder.java:268)
          at javax.xml.datatype.DatatypeFactory.newInstance(DatatypeFactory.java:145)
          at com.thoughtworks.xstream.converters.extended.DurationConverter$1.getFactory(DurationConverter.java:39)
          at com.thoughtworks.xstream.converters.extended.DurationConverter.<init>(DurationConverter.java:36)
          ... 31 more
      Caused by: java.lang.InterruptedException
          at java.lang.Object.wait(Native Method)
          at hudson.remoting.Request.call(Request.java:177)
          at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286)
          ... 44 more
       
      May 04, 2018 9:16:41 AM hudson.remoting.UserRequest perform
      WARNING: LinkageError while performing UserRequest:hudson.Launcher$RemoteLauncher$KillTask@14f02673
      java.lang.ExceptionInInitializerError
          at hudson.model.Computer.<clinit>(Computer.java:1761)
          at hudson.util.ProcessTree.getKillers(ProcessTree.java:157)
          at hudson.util.ProcessTree$OSProcess.killByKiller(ProcessTree.java:220)
          at hudson.util.ProcessTree$UnixProcess.kill(ProcessTree.java:662)
          at hudson.util.ProcessTree$UnixProcess.killRecursively(ProcessTree.java:670)
          at hudson.util.ProcessTree$UnixProcess.killRecursively(ProcessTree.java:669)
          at hudson.util.ProcessTree$UnixProcess.killRecursively(ProcessTree.java:669)
          at hudson.util.ProcessTree$Unix.killAll(ProcessTree.java:591)
          at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1090)
          at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1081)
          at hudson.remoting.UserRequest.perform(UserRequest.java:212)
          at hudson.remoting.UserRequest.perform(UserRequest.java:54)
          at hudson.remoting.Request$2.run(Request.java:369)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at java.lang.Thread.run(Thread.java:748)
      Caused by: com.thoughtworks.xstream.InitializationException: Could not instantiate converter : com.thoughtworks.xstream.converters.extended.DurationConverter : null
          at com.thoughtworks.xstream.XStream.registerConverterDynamically(XStream.java:904)
          at com.thoughtworks.xstream.XStream.setupConverters(XStream.java:867)
          at com.thoughtworks.xstream.XStream.<init>(XStream.java:574)
          at com.thoughtworks.xstream.XStream.<init>(XStream.java:496)
          at com.thoughtworks.xstream.XStream.<init>(XStream.java:465)
          at com.thoughtworks.xstream.XStream.<init>(XStream.java:411)
          at com.thoughtworks.xstream.XStream.<init>(XStream.java:378)
          at hudson.util.XStream2.<init>(XStream2.java:112)
          at jenkins.model.Jenkins.<clinit>(Jenkins.java:5103)
          ... 18 more
      Caused by: java.lang.reflect.InvocationTargetException
          at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
          at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
          at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
          at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
          at com.thoughtworks.xstream.XStream.registerConverterDynamically(XStream.java:897)
          ... 26 more
      Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException
          at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:299)
          at com.sun.proxy.$Proxy5.getResources2(Unknown Source)
          at hudson.remoting.RemoteClassLoader.findResources(RemoteClassLoader.java:481)
          at java.lang.ClassLoader.getResources(ClassLoader.java:1142)
          at java.util.ServiceLoader$LazyIterator.hasNextService(ServiceLoader.java:348)
          at java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:393)
          at java.util.ServiceLoader$1.hasNext(ServiceLoader.java:474)
          at javax.xml.datatype.FactoryFinder$1.run(FactoryFinder.java:296)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.xml.datatype.FactoryFinder.findServiceProvider(FactoryFinder.java:292)
          at javax.xml.datatype.FactoryFinder.find(FactoryFinder.java:268)
          at javax.xml.datatype.DatatypeFactory.newInstance(DatatypeFactory.java:145)
          at com.thoughtworks.xstream.converters.extended.DurationConverter$1.getFactory(DurationConverter.java:39)
          at com.thoughtworks.xstream.converters.extended.DurationConverter.<init>(DurationConverter.java:36)
          ... 31 more
      Caused by: java.lang.InterruptedException
          at java.lang.Object.wait(Native Method)
          at hudson.remoting.Request.call(Request.java:177)
          at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286)
          ... 44 more

       

       

        Attachments

          Activity

          Hide
          markewaite Mark Waite added a comment -

          Any chance that the agent process is running in Java 9, instead of Java 8? Jenkins masters and Jenkins agents are not supported with Java 9. The xstream code has known problems with the Java 9 module system.

          Show
          markewaite Mark Waite added a comment - Any chance that the agent process is running in Java 9, instead of Java 8? Jenkins masters and Jenkins agents are not supported with Java 9. The xstream code has known problems with the Java 9 module system.
          Hide
          tdudala tdudala added a comment -

          Nope, both slave and master are on Java 1.8.0_131 

          Show
          tdudala tdudala added a comment - Nope, both slave and master are on Java 1.8.0_131 
          Hide
          jthompson Jeff Thompson added a comment -

          I'm at a loss to explain this one.

          I've seen a few other reports where people using pipeline were getting an InterruptedException. I haven't yet seen any indication of what caused it. Maybe an interaction between different plugins.

          I see timestamps in the agent logs but none in the master. Do these different events and logs match up?

          My only suggestion is to try and isolate or reproduce the problem somehow. 

          Have you seen any changes in the behavior or the frequency?

          Show
          jthompson Jeff Thompson added a comment - I'm at a loss to explain this one. I've seen a few other reports where people using pipeline were getting an InterruptedException. I haven't yet seen any indication of what caused it. Maybe an interaction between different plugins. I see timestamps in the agent logs but none in the master. Do these different events and logs match up? My only suggestion is to try and isolate or reproduce the problem somehow.  Have you seen any changes in the behavior or the frequency?
          Hide
          romanz Roman Zwi added a comment -

          Is the correct version of slave.jar running on the slave? You can download it from <yourjenkinsurl>/jnlpJars/slave.jar
          You can use the Versions Node Monitors plugin to have it checked automatically. It also adds the versions to the node list.

          Show
          romanz Roman Zwi added a comment - Is the correct version of slave.jar running on the slave? You can download it from <yourjenkinsurl>/jnlpJars/slave.jar You can use the Versions Node Monitors plugin to have it checked automatically. It also adds the versions to the node list.
          Hide
          jthompson Jeff Thompson added a comment -

          Closing for lack of sufficient diagnostics and information to reproduce after no response for quite a while.

          Show
          jthompson Jeff Thompson added a comment - Closing for lack of sufficient diagnostics and information to reproduce after no response for quite a while.

            People

            • Assignee:
              Unassigned
              Reporter:
              tdudala tdudala
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: