Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Cannot Reproduce
    • Component/s: remoting
    • Labels:
      None
    • Similar Issues:

      Description

      Builds are randomly failing when assigned to some linux agents. Those agents are part of a fleet whose other instances work. Usually disconnecting then reconnecting to the instance fixes the problem. The builds that experience this issue fail pretty much right away.

      java.lang.InterruptedException
      	at java.lang.Object.wait(Native Method)
      	at hudson.remoting.Request.call(Request.java:177)
      	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286)
      Caused: hudson.remoting.RemotingSystemException
      	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:299)
      	at com.sun.proxy.$Proxy5.fetch(Unknown Source)
      	at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:315)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      	at java.lang.ClassLoader.defineClass1(Native Method)
      	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
      	at java.lang.ClassLoader.defineClass(ClassLoader.java:642)
      	at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:383)
      	at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:307)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      	at java.lang.Class.getDeclaredMethods0(Native Method)
      	at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
      	at java.lang.Class.getDeclaredMethod(Class.java:2128)
      	at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1629)
      	at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:79)
      	at java.io.ObjectStreamClass$3.run(ObjectStreamClass.java:520)
      	at java.io.ObjectStreamClass$3.run(ObjectStreamClass.java:494)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:494)
      	at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:391)
      	at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:681)
      	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1885)
      	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
      	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
      	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
      	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
      	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
      	at hudson.remoting.UserRequest.deserialize(UserRequest.java:291)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:190)
      Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to i-0ee97de5eafef4606
      		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)
      		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
      		at hudson.remoting.Channel.call(Channel.java:957)
      		at hudson.FilePath.act(FilePath.java:1072)
      		at hudson.FilePath.act(FilePath.java:1061)
      		at hudson.FilePath.deleteRecursive(FilePath.java:1268)
      		at org.jenkinsci.plugins.workflow.steps.DeleteDirStep$Execution.run(DeleteDirStep.java:77)
      		at org.jenkinsci.plugins.workflow.steps.DeleteDirStep$Execution.run(DeleteDirStep.java:69)
      		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
      		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      Caused: java.lang.Error: Failed to deserialize the Callable object.
      	at hudson.remoting.UserRequest.perform(UserRequest.java:196)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
      	at hudson.remoting.Request$2.run(Request.java:369)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      Caused: java.io.IOException: Remote call on i-0ee97de5eafef4606 failed
      	at hudson.remoting.Channel.call(Channel.java:963)
      	at hudson.FilePath.act(FilePath.java:1072)
      	at hudson.FilePath.act(FilePath.java:1061)
      	at hudson.FilePath.deleteRecursive(FilePath.java:1268)
      	at org.jenkinsci.plugins.workflow.steps.DeleteDirStep$Execution.run(DeleteDirStep.java:77)
      	at org.jenkinsci.plugins.workflow.steps.DeleteDirStep$Execution.run(DeleteDirStep.java:69)
      	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Finished: FAILURE
      

        Attachments

          Activity

          Hide
          jzila John Zila added a comment -

          Ping. This is making Jenkins unusable.

          Show
          jzila John Zila added a comment - Ping. This is making Jenkins unusable.
          Hide
          jthompson Jeff Thompson added a comment -

          Unfortunately, there isn't enough information in this report to make it actionable. Random failures can be difficult to diagnose when you have access to the systems and can examine differences in status and behaviors.

          One of the first things to check is that you have compatible versions of both Java and the agent (slave) jar. Given the randomness, this seems unlikely but it is essential to validate it.

          Try to isolate anything you can about what is different between the successes and failures. Do the failures always occur with the same action? (In this case it is DeleteDirStep.) Can you observe any differences in the types of systems where it succeeds or fails? Are there differences in the job configurations?

          Are you up-to-date with Jenkins and plugin versions? Are your agents up-to-date?

          Many times unexplained failures result from some combination of plugins. Tracking them down may require running experiments with disabling one or more plugins.

          At a minimum, this issue report needs to include the version of Jenkins you are using, along with plugins and their versions. Perhaps others have seen some similar behavior and may be to provide ideas.

          Show
          jthompson Jeff Thompson added a comment - Unfortunately, there isn't enough information in this report to make it actionable. Random failures can be difficult to diagnose when you have access to the systems and can examine differences in status and behaviors. One of the first things to check is that you have compatible versions of both Java and the agent (slave) jar. Given the randomness, this seems unlikely but it is essential to validate it. Try to isolate anything you can about what is different between the successes and failures. Do the failures always occur with the same action? (In this case it is DeleteDirStep.) Can you observe any differences in the types of systems where it succeeds or fails? Are there differences in the job configurations? Are you up-to-date with Jenkins and plugin versions? Are your agents up-to-date? Many times unexplained failures result from some combination of plugins. Tracking them down may require running experiments with disabling one or more plugins. At a minimum, this issue report needs to include the version of Jenkins you are using, along with plugins and their versions. Perhaps others have seen some similar behavior and may be to provide ideas.
          Hide
          jzila John Zila added a comment -

          This seems to not be happening any more. I'm going to close the issue.

          Show
          jzila John Zila added a comment - This seems to not be happening any more. I'm going to close the issue.

            People

            • Assignee:
              jthompson Jeff Thompson
              Reporter:
              jzila John Zila
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: