Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-8856

StreamCorruptedException

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Randomly some builds fails on ssh-slaves because of a StreamCorruptedException.

      It was not happening at all a couple of version ago.

      Here is the log:

      Construction à distance sur XXXXXXX
      FATAL: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid type code: 4B
      hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid type code: 4B
      at hudson.remoting.Request.call(Request.java:137)
      at hudson.remoting.Channel.call(Channel.java:629)
      at hudson.FilePath.act(FilePath.java:745)
      at hudson.FilePath.act(FilePath.java:738)
      at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:683)
      at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:632)
      at hudson.model.AbstractProject.checkout(AbstractProject.java:1171)
      at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:501)
      at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:415)
      at hudson.model.Run.run(Run.java:1362)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:145)
      Caused by: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid type code: 4B
      at hudson.remoting.Request.abort(Request.java:257)
      at hudson.remoting.Channel.terminate(Channel.java:680)
      at hudson.remoting.Channel$ReaderThread.run(Channel.java:971)
      Caused by: java.io.StreamCorruptedException: invalid type code: 4B
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1355)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
      at hudson.remoting.Channel$ReaderThread.run(Channel.java:947)

        Attachments

          Issue Links

            Activity

            ebannn i b created issue -
            Hide
            hostalp Petr H added a comment -

            I'm getting sort of similar stuff in mixed 64/32 bit environment.

            Master is running at x86_64 Linux OS (Tomcat 6.0.32 with locally built tcnative 1.1.20), Java 6u26 64 bit
            SSH slave is running at the same OS, and here I tried several Java releases and so far it seems to be reproducible only at 32 bit JVM. And even then it occurs randomly.
            Slave agent is running at Java 6u26 as well, while the actual builds (Maven 2.2.1) are (and have to be) done at Java 1.5 (1.5u30). When I use 32 bit JVM here (in order to reduce memory footprint) the StreamCorruptedException randomly occurs during build. When I Switch everything to 64 bit I'm not able to reproduce the issue (at least not for now - and I tried hard).

            In my case the particular issue is like:

            java.lang.reflect.UndeclaredThrowableException
            at hudson.remoting.$Proxy1.fetch2(Unknown Source)
            at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:122)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
            at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
            at hudson.maven.reporters.MavenArtifactArchiver.postBuild(MavenArtifactArchiver.java:99)
            at hudson.maven.MavenModuleSetBuild$Builder.postModule(MavenModuleSetBuild.java:1051)
            at hudson.maven.MavenBuilder$Adapter.fireLeaveModule(MavenBuilder.java:353)
            at hudson.maven.MavenBuilder$Adapter.postBuild(MavenBuilder.java:311)
            at org.apache.maven.lifecycle.LifecycleExecutorInterceptor.execute(LifecycleExecutorInterceptor.java:68)
            at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:328)
            at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:138)
            at org.apache.maven.cli.MavenCli.main(MavenCli.java:362)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:592)
            at org.codehaus.classworlds.Launcher.launchEnhanced(Launcher.java:315)
            at org.codehaus.classworlds.Launcher.launch(Launcher.java:255)
            at hudson.maven.agent.Main.launch(Main.java:185)
            at hudson.maven.MavenBuilder.call(MavenBuilder.java:164)
            at hudson.maven.MavenModuleSetBuild$Builder.call(MavenModuleSetBuild.java:1001)
            at hudson.maven.MavenModuleSetBuild$Builder.call(MavenModuleSetBuild.java:932)
            at hudson.remoting.UserRequest.perform(UserRequest.java:118)
            at hudson.remoting.UserRequest.perform(UserRequest.java:48)
            at hudson.remoting.Request$2.run(Request.java:283)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
            at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
            at java.util.concurrent.FutureTask.run(FutureTask.java:123)
            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676)
            at java.lang.Thread.run(Thread.java:637)
            Caused by: hudson.remoting.ChannelClosedException: channel is already closed
            at hudson.remoting.Channel.send(Channel.java:486)
            at hudson.remoting.Request.call(Request.java:110)
            at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:160)
            ... 32 more
            Caused by: java.io.StreamCorruptedException
            at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1331)
            at java.io.ObjectInputStream.readObject(ObjectInputStream.java:347)
            at hudson.remoting.Channel$ReaderThread.run(Channel.java:1019)

            But sometimes I'm getting strange ClasFormatError ones such as:
            1.
            java.lang.ClassFormatError: LocalVariableTable has wrong length in class file hudson/maven/reporters/MavenArtifact
            at java.lang.ClassLoader.defineClass1(Native Method)
            at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
            at java.lang.ClassLoader.defineClass(ClassLoader.java:466)
            at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:151)
            at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:131)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
            at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
            at hudson.maven.reporters.MavenArtifactArchiver.postBuild(MavenArtifactArchiver.java:99)
            at hudson.maven.MavenModuleSetBuild$Builder.postModule(MavenModuleSetBuild.java:1051)
            at hudson.maven.MavenBuilder$Adapter.fireLeaveModule(MavenBuilder.java:353)
            at hudson.maven.MavenBuilder$Adapter.endModule(MavenBuilder.java:318)
            at org.apache.maven.lifecycle.LifecycleExecutorInterceptor$EventMonitorImpl.endEvent(LifecycleExecutorInterceptor.java:92)
            at org.apache.maven.monitor.event.DefaultEventDispatcher.dispatchEnd(DefaultEventDispatcher.java:54)
            at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeTaskSegments(DefaultLifecycleExecutor.java:360)
            at org.apache.maven.lifecycle.DefaultLifecycleExecutor.execute(DefaultLifecycleExecutor.java:180)
            at org.apache.maven.lifecycle.LifecycleExecutorInterceptor.execute(LifecycleExecutorInterceptor.java:65)
            at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:328)
            at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:138)
            at org.apache.maven.cli.MavenCli.main(MavenCli.java:362)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:592)
            at org.codehaus.classworlds.Launcher.launchEnhanced(Launcher.java:315)
            at org.codehaus.classworlds.Launcher.launch(Launcher.java:255)
            at hudson.maven.agent.Main.launch(Main.java:185)
            at hudson.maven.MavenBuilder.call(MavenBuilder.java:164)
            at hudson.maven.MavenModuleSetBuild$Builder.call(MavenModuleSetBuild.java:1001)
            at hudson.maven.MavenModuleSetBuild$Builder.call(MavenModuleSetBuild.java:932)
            at hudson.remoting.UserRequest.perform(UserRequest.java:118)
            at hudson.remoting.UserRequest.perform(UserRequest.java:48)
            at hudson.remoting.Request$2.run(Request.java:283)

            2.
            java.lang.ClassFormatError: Illegal UTF8 string in constant pool in class file hudson/maven/reporters/MavenArtifact
            at java.lang.ClassLoader.defineClass1(Native Method)
            at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
            at java.lang.ClassLoader.defineClass(ClassLoader.java:466)
            at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:151)
            at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:131)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
            at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
            at hudson.maven.reporters.MavenArtifactArchiver.postBuild(MavenArtifactArchiver.java:99)
            at hudson.maven.MavenModuleSetBuild$Builder.postModule(MavenModuleSetBuild.java:1051)
            at hudson.maven.MavenBuilder$Adapter.fireLeaveModule(MavenBuilder.java:353)
            at hudson.maven.MavenBuilder$Adapter.endModule(MavenBuilder.java:318)
            at org.apache.maven.lifecycle.LifecycleExecutorInterceptor$EventMonitorImpl.endEvent(LifecycleExecutorInterceptor.java:92)
            at org.apache.maven.monitor.event.DefaultEventDispatcher.dispatchEnd(DefaultEventDispatcher.java:54)
            at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeTaskSegments(DefaultLifecycleExecutor.java:360)
            at org.apache.maven.lifecycle.DefaultLifecycleExecutor.execute(DefaultLifecycleExecutor.java:180)
            at org.apache.maven.lifecycle.LifecycleExecutorInterceptor.execute(LifecycleExecutorInterceptor.java:65)
            at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:328)
            at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:138)
            at org.apache.maven.cli.MavenCli.main(MavenCli.java:362)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:592)
            at org.codehaus.classworlds.Launcher.launchEnhanced(Launcher.java:315)
            at org.codehaus.classworlds.Launcher.launch(Launcher.java:255)
            at hudson.maven.agent.Main.launch(Main.java:185)
            at hudson.maven.MavenBuilder.call(MavenBuilder.java:164)
            at hudson.maven.MavenModuleSetBuild$Builder.call(MavenModuleSetBuild.java:1001)
            at hudson.maven.MavenModuleSetBuild$Builder.call(MavenModuleSetBuild.java:932)
            at hudson.remoting.UserRequest.perform(UserRequest.java:118)
            at hudson.remoting.UserRequest.perform(UserRequest.java:48)
            at hudson.remoting.Request$2.run(Request.java:283)

            3. or this another one
            java.io.UTFDataFormatException
            at java.io.ObjectInputStream$BlockDataInputStream.readUTFSpan(ObjectInputStream.java:3008)
            at java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:2952)
            at java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:2765)
            at java.io.ObjectInputStream.readUTF(ObjectInputStream.java:1031)
            at java.io.ObjectStreamClass.readNonProxy(ObjectStreamClass.java:631)
            at java.io.ObjectInputStream.readClassDescriptor(ObjectInputStream.java:788)
            at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1533)
            at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1465)
            at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1698)
            at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1304)
            at java.io.ObjectInputStream.readObject(ObjectInputStream.java:347)
            at hudson.remoting.Channel$ReaderThread.run(Channel.java:1019)

            All the above comes from Jenkins 1.417 (however 1.418 makes no difference).

            Note: I've got one more 64 bit slave machine with almost identical setup where the issue seems to never occur. I'm still trying to figure out why, but have found nothing interesting so far. OS and JDKs are the same at both machines.

            One side note (I may create a separate issue for that if it's possible to fix it): Another strange thing I noticed when building at Java 1.5 is that if I use Tomcat 7 (7.0.16) I can't get it to work. After some tracing it seems that builder process at slave tries to load few javax.servlet classes (ServletException and few others) via Jenkins remoting, these seem to be coming from the Tomcat 7 itself where they're built for Java 6 and thus I'm getting "java.lang.UnsupportedClassVersionError: Bad version number in .class file". That's quite unfortunate and limiting.

            Show
            hostalp Petr H added a comment - I'm getting sort of similar stuff in mixed 64/32 bit environment. Master is running at x86_64 Linux OS (Tomcat 6.0.32 with locally built tcnative 1.1.20), Java 6u26 64 bit SSH slave is running at the same OS, and here I tried several Java releases and so far it seems to be reproducible only at 32 bit JVM. And even then it occurs randomly. Slave agent is running at Java 6u26 as well, while the actual builds (Maven 2.2.1) are (and have to be) done at Java 1.5 (1.5u30). When I use 32 bit JVM here (in order to reduce memory footprint) the StreamCorruptedException randomly occurs during build. When I Switch everything to 64 bit I'm not able to reproduce the issue (at least not for now - and I tried hard). In my case the particular issue is like: java.lang.reflect.UndeclaredThrowableException at hudson.remoting.$Proxy1.fetch2(Unknown Source) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:122) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at java.lang.ClassLoader.loadClass(ClassLoader.java:252) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) at hudson.maven.reporters.MavenArtifactArchiver.postBuild(MavenArtifactArchiver.java:99) at hudson.maven.MavenModuleSetBuild$Builder.postModule(MavenModuleSetBuild.java:1051) at hudson.maven.MavenBuilder$Adapter.fireLeaveModule(MavenBuilder.java:353) at hudson.maven.MavenBuilder$Adapter.postBuild(MavenBuilder.java:311) at org.apache.maven.lifecycle.LifecycleExecutorInterceptor.execute(LifecycleExecutorInterceptor.java:68) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:328) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:138) at org.apache.maven.cli.MavenCli.main(MavenCli.java:362) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:592) at org.codehaus.classworlds.Launcher.launchEnhanced(Launcher.java:315) at org.codehaus.classworlds.Launcher.launch(Launcher.java:255) at hudson.maven.agent.Main.launch(Main.java:185) at hudson.maven.MavenBuilder.call(MavenBuilder.java:164) at hudson.maven.MavenModuleSetBuild$Builder.call(MavenModuleSetBuild.java:1001) at hudson.maven.MavenModuleSetBuild$Builder.call(MavenModuleSetBuild.java:932) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:283) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269) at java.util.concurrent.FutureTask.run(FutureTask.java:123) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676) at java.lang.Thread.run(Thread.java:637) Caused by: hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:486) at hudson.remoting.Request.call(Request.java:110) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:160) ... 32 more Caused by: java.io.StreamCorruptedException at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1331) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:347) at hudson.remoting.Channel$ReaderThread.run(Channel.java:1019) But sometimes I'm getting strange ClasFormatError ones such as: 1. java.lang.ClassFormatError: LocalVariableTable has wrong length in class file hudson/maven/reporters/MavenArtifact at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:621) at java.lang.ClassLoader.defineClass(ClassLoader.java:466) at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:151) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:131) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at java.lang.ClassLoader.loadClass(ClassLoader.java:252) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) at hudson.maven.reporters.MavenArtifactArchiver.postBuild(MavenArtifactArchiver.java:99) at hudson.maven.MavenModuleSetBuild$Builder.postModule(MavenModuleSetBuild.java:1051) at hudson.maven.MavenBuilder$Adapter.fireLeaveModule(MavenBuilder.java:353) at hudson.maven.MavenBuilder$Adapter.endModule(MavenBuilder.java:318) at org.apache.maven.lifecycle.LifecycleExecutorInterceptor$EventMonitorImpl.endEvent(LifecycleExecutorInterceptor.java:92) at org.apache.maven.monitor.event.DefaultEventDispatcher.dispatchEnd(DefaultEventDispatcher.java:54) at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeTaskSegments(DefaultLifecycleExecutor.java:360) at org.apache.maven.lifecycle.DefaultLifecycleExecutor.execute(DefaultLifecycleExecutor.java:180) at org.apache.maven.lifecycle.LifecycleExecutorInterceptor.execute(LifecycleExecutorInterceptor.java:65) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:328) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:138) at org.apache.maven.cli.MavenCli.main(MavenCli.java:362) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:592) at org.codehaus.classworlds.Launcher.launchEnhanced(Launcher.java:315) at org.codehaus.classworlds.Launcher.launch(Launcher.java:255) at hudson.maven.agent.Main.launch(Main.java:185) at hudson.maven.MavenBuilder.call(MavenBuilder.java:164) at hudson.maven.MavenModuleSetBuild$Builder.call(MavenModuleSetBuild.java:1001) at hudson.maven.MavenModuleSetBuild$Builder.call(MavenModuleSetBuild.java:932) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:283) 2. java.lang.ClassFormatError: Illegal UTF8 string in constant pool in class file hudson/maven/reporters/MavenArtifact at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:621) at java.lang.ClassLoader.defineClass(ClassLoader.java:466) at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:151) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:131) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at java.lang.ClassLoader.loadClass(ClassLoader.java:252) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) at hudson.maven.reporters.MavenArtifactArchiver.postBuild(MavenArtifactArchiver.java:99) at hudson.maven.MavenModuleSetBuild$Builder.postModule(MavenModuleSetBuild.java:1051) at hudson.maven.MavenBuilder$Adapter.fireLeaveModule(MavenBuilder.java:353) at hudson.maven.MavenBuilder$Adapter.endModule(MavenBuilder.java:318) at org.apache.maven.lifecycle.LifecycleExecutorInterceptor$EventMonitorImpl.endEvent(LifecycleExecutorInterceptor.java:92) at org.apache.maven.monitor.event.DefaultEventDispatcher.dispatchEnd(DefaultEventDispatcher.java:54) at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeTaskSegments(DefaultLifecycleExecutor.java:360) at org.apache.maven.lifecycle.DefaultLifecycleExecutor.execute(DefaultLifecycleExecutor.java:180) at org.apache.maven.lifecycle.LifecycleExecutorInterceptor.execute(LifecycleExecutorInterceptor.java:65) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:328) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:138) at org.apache.maven.cli.MavenCli.main(MavenCli.java:362) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:592) at org.codehaus.classworlds.Launcher.launchEnhanced(Launcher.java:315) at org.codehaus.classworlds.Launcher.launch(Launcher.java:255) at hudson.maven.agent.Main.launch(Main.java:185) at hudson.maven.MavenBuilder.call(MavenBuilder.java:164) at hudson.maven.MavenModuleSetBuild$Builder.call(MavenModuleSetBuild.java:1001) at hudson.maven.MavenModuleSetBuild$Builder.call(MavenModuleSetBuild.java:932) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:283) 3. or this another one java.io.UTFDataFormatException at java.io.ObjectInputStream$BlockDataInputStream.readUTFSpan(ObjectInputStream.java:3008) at java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:2952) at java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:2765) at java.io.ObjectInputStream.readUTF(ObjectInputStream.java:1031) at java.io.ObjectStreamClass.readNonProxy(ObjectStreamClass.java:631) at java.io.ObjectInputStream.readClassDescriptor(ObjectInputStream.java:788) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1533) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1465) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1698) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1304) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:347) at hudson.remoting.Channel$ReaderThread.run(Channel.java:1019) All the above comes from Jenkins 1.417 (however 1.418 makes no difference). Note: I've got one more 64 bit slave machine with almost identical setup where the issue seems to never occur. I'm still trying to figure out why, but have found nothing interesting so far. OS and JDKs are the same at both machines. One side note (I may create a separate issue for that if it's possible to fix it): Another strange thing I noticed when building at Java 1.5 is that if I use Tomcat 7 (7.0.16) I can't get it to work. After some tracing it seems that builder process at slave tries to load few javax.servlet classes (ServletException and few others) via Jenkins remoting, these seem to be coming from the Tomcat 7 itself where they're built for Java 6 and thus I'm getting "java.lang.UnsupportedClassVersionError: Bad version number in .class file". That's quite unfortunate and limiting.
            Hide
            kohsuke Kohsuke Kawaguchi added a comment -

            The SSH slaves plugin set up the connection via stdin/stdout of the Java process. We replace System.out, so that Java programs writing to System.out won't corrupt the stream, but if some native code in JVM writes to the stdout (file handle 1), then it can corrupt the data stream. The class loading failure implies that this is what's going on. Some JVM options are known to do this, too.

            I guess I should do dup and close system calls to avoid using the stdout so that it won't corrupt the stream. I should also introduce the optional framing with checksum to detect this more eagerly.

            Show
            kohsuke Kohsuke Kawaguchi added a comment - The SSH slaves plugin set up the connection via stdin/stdout of the Java process. We replace System.out, so that Java programs writing to System.out won't corrupt the stream, but if some native code in JVM writes to the stdout (file handle 1), then it can corrupt the data stream. The class loading failure implies that this is what's going on. Some JVM options are known to do this, too. I guess I should do dup and close system calls to avoid using the stdout so that it won't corrupt the stream. I should also introduce the optional framing with checksum to detect this more eagerly.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Kohsuke Kawaguchi
            Path:
            changelog.html
            core/src/main/java/hudson/util/jna/GNUCLibrary.java
            core/src/main/java/jenkins/slaves/StandardOutputSwapper.java
            pom.xml
            http://jenkins-ci.org/commit/jenkins/3f4d01ec15ee40f94e861e9e70403ac6b16c1ed3
            Log:
            JENKINS-8856 evacuate stdout at OS level to avoid interference with
            JVM.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html core/src/main/java/hudson/util/jna/GNUCLibrary.java core/src/main/java/jenkins/slaves/StandardOutputSwapper.java pom.xml http://jenkins-ci.org/commit/jenkins/3f4d01ec15ee40f94e861e9e70403ac6b16c1ed3 Log: JENKINS-8856 evacuate stdout at OS level to avoid interference with JVM.
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #1076
            JENKINS-8856 evacuate stdout at OS level to avoid interference with

            Kohsuke Kawaguchi : 3f4d01ec15ee40f94e861e9e70403ac6b16c1ed3
            Files :

            • pom.xml
            • core/src/main/java/jenkins/slaves/StandardOutputSwapper.java
            • core/src/main/java/hudson/util/jna/GNUCLibrary.java
            • changelog.html
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #1076 JENKINS-8856 evacuate stdout at OS level to avoid interference with Kohsuke Kawaguchi : 3f4d01ec15ee40f94e861e9e70403ac6b16c1ed3 Files : pom.xml core/src/main/java/jenkins/slaves/StandardOutputSwapper.java core/src/main/java/hudson/util/jna/GNUCLibrary.java changelog.html
            Hide
            kohsuke Kohsuke Kawaguchi added a comment -

            I won't be doing framing. See the related remoting commits for why it can induce the reader block.

            So I'm marking this issue as closed.

            Show
            kohsuke Kohsuke Kawaguchi added a comment - I won't be doing framing. See the related remoting commits for why it can induce the reader block. So I'm marking this issue as closed.
            kohsuke Kohsuke Kawaguchi made changes -
            Field Original Value New Value
            Status Open [ 1 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Kohsuke Kawaguchi
            Path:
            changelog.html
            core/src/main/java/hudson/util/jna/GNUCLibrary.java
            core/src/main/java/jenkins/slaves/StandardOutputSwapper.java
            pom.xml
            http://jenkins-ci.org/commit/jenkins/3f4d01ec15ee40f94e861e9e70403ac6b16c1ed3
            Log:
            JENKINS-8856 evacuate stdout at OS level to avoid interference with
            JVM.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html core/src/main/java/hudson/util/jna/GNUCLibrary.java core/src/main/java/jenkins/slaves/StandardOutputSwapper.java pom.xml http://jenkins-ci.org/commit/jenkins/3f4d01ec15ee40f94e861e9e70403ac6b16c1ed3 Log: JENKINS-8856 evacuate stdout at OS level to avoid interference with JVM.
            Hide
            morriste Stephen Morrison added a comment -

            Is this likely to have affected ordering of stdout and sterr in the log? We are seeing a problem now where output from an SSH linux slave has different ordering of stdout and stderr in the Jenkins log as opposed to when you run the build manually logged into the machine.

            Show
            morriste Stephen Morrison added a comment - Is this likely to have affected ordering of stdout and sterr in the log? We are seeing a problem now where output from an SSH linux slave has different ordering of stdout and stderr in the Jenkins log as opposed to when you run the build manually logged into the machine.
            Hide
            ingunawa Indra Gunawan added a comment -

            We are seeing this error on Jenkins 1.455

            INFO: Attempting to reconnect sjc-bld87-lnx
            Apr 24, 2012 2:22:43 PM hudson.remoting.Channel$ReaderThread run
            SEVERE: I/O error in channel sjc-bld87-lnx
            java.io.StreamCorruptedException: invalid type code: 23
            at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1355)
            at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
            at hudson.remoting.Channel$ReaderThread.run(Channel.java:1127)
            Apr 24, 2012 2:22:45 PM hudson.slaves.CommandLauncher launchINFO: Attempting to reconnect sjc-bld87-lnx
            Apr 24, 2012 2:22:43 PM hudson.remoting.Channel$ReaderThread run
            SEVERE: I/O error in channel sjc-bld87-lnx
            java.io.StreamCorruptedException: invalid type code: 23
            at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1355)
            at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
            at hudson.remoting.Channel$ReaderThread.run(Channel.java:1127)
            Apr 24, 2012 2:22:45 PM hudson.slaves.CommandLauncher launch

            The build is not starting and stuck on the slave.

            -Indra

            Show
            ingunawa Indra Gunawan added a comment - We are seeing this error on Jenkins 1.455 INFO: Attempting to reconnect sjc-bld87-lnx Apr 24, 2012 2:22:43 PM hudson.remoting.Channel$ReaderThread run SEVERE: I/O error in channel sjc-bld87-lnx java.io.StreamCorruptedException: invalid type code: 23 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1355) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350) at hudson.remoting.Channel$ReaderThread.run(Channel.java:1127) Apr 24, 2012 2:22:45 PM hudson.slaves.CommandLauncher launchINFO: Attempting to reconnect sjc-bld87-lnx Apr 24, 2012 2:22:43 PM hudson.remoting.Channel$ReaderThread run SEVERE: I/O error in channel sjc-bld87-lnx java.io.StreamCorruptedException: invalid type code: 23 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1355) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350) at hudson.remoting.Channel$ReaderThread.run(Channel.java:1127) Apr 24, 2012 2:22:45 PM hudson.slaves.CommandLauncher launch The build is not starting and stuck on the slave. -Indra
            Hide
            ingunawa Indra Gunawan added a comment -

            Is it possible that this is being looked at again? We are getting the Channel slave termination i.e. build can't proceed on slave and hang.

            We are using Jenkins 1.455 winstone container with JDK 1.6.0_29 64 bit LNX.

            We are using "Launching command on master" to connect to slave. Basically we are using SSH to run a shell script on linux slave with 32 bit JDK 6.0.12 with this option:
            -Xms2048m -Xmx2048m

            I am at Cisco. A while back, you and Sasha dropped by at our building to meet Max Spring and our group.

            Thank you.
            -Indra

            Show
            ingunawa Indra Gunawan added a comment - Is it possible that this is being looked at again? We are getting the Channel slave termination i.e. build can't proceed on slave and hang. We are using Jenkins 1.455 winstone container with JDK 1.6.0_29 64 bit LNX. We are using "Launching command on master" to connect to slave. Basically we are using SSH to run a shell script on linux slave with 32 bit JDK 6.0.12 with this option: -Xms2048m -Xmx2048m I am at Cisco. A while back, you and Sasha dropped by at our building to meet Max Spring and our group. Thank you. -Indra
            ingunawa Indra Gunawan made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            Hide
            scott_sandler Scott Sandler added a comment -

            I ran into this issue today. Turns out the problem was that we had added this java option to the slave start command: -XX:+PrintGCDetails. Removing that fixed the issue.

            Show
            scott_sandler Scott Sandler added a comment - I ran into this issue today. Turns out the problem was that we had added this java option to the slave start command: -XX:+PrintGCDetails. Removing that fixed the issue.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Kohsuke Kawaguchi
            Path:
            pom.xml
            src/main/java/hudson/remoting/ClassicCommandTransport.java
            src/main/java/hudson/remoting/DiagnosedStreamCorruptionException.java
            src/test/java/hudson/remoting/DiagnosedStreamCorruptionExceptionTest.java
            http://jenkins-ci.org/commit/remoting/68be3b1741d8e9e2da8e31491b1b558a8cdbd33b
            Log:
            JENKINS-8856

            Added additional diagnostic code that will try to read ahead and provide
            what's ahead in the input stream.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: pom.xml src/main/java/hudson/remoting/ClassicCommandTransport.java src/main/java/hudson/remoting/DiagnosedStreamCorruptionException.java src/test/java/hudson/remoting/DiagnosedStreamCorruptionExceptionTest.java http://jenkins-ci.org/commit/remoting/68be3b1741d8e9e2da8e31491b1b558a8cdbd33b Log: JENKINS-8856 Added additional diagnostic code that will try to read ahead and provide what's ahead in the input stream.
            Hide
            kohsuke Kohsuke Kawaguchi added a comment -

            The above commit will read ahead and dump the content of the stream when a StreamCorruptedException is encountered. The fix is in remoting 2.25 and Jenkins 1.521.

            If anyone is still seeing this problem, please use these versions and report the complete stack trace, which will include the "read ahead" section.

            Show
            kohsuke Kohsuke Kawaguchi added a comment - The above commit will read ahead and dump the content of the stream when a StreamCorruptedException is encountered. The fix is in remoting 2.25 and Jenkins 1.521. If anyone is still seeing this problem, please use these versions and report the complete stack trace, which will include the "read ahead" section.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Kohsuke Kawaguchi
            Path:
            changelog.html
            pom.xml
            http://jenkins-ci.org/commit/jenkins/159566351bc5aa6b83d400c2f34f508cf05f1a8f
            Log:
            JENKINS-8856

            Integrated the new version of remoting

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html pom.xml http://jenkins-ci.org/commit/jenkins/159566351bc5aa6b83d400c2f34f508cf05f1a8f Log: JENKINS-8856 Integrated the new version of remoting
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #2648
            JENKINS-8856 (Revision 159566351bc5aa6b83d400c2f34f508cf05f1a8f)

            Result = SUCCESS
            kohsuke : 159566351bc5aa6b83d400c2f34f508cf05f1a8f
            Files :

            • pom.xml
            • changelog.html
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #2648 JENKINS-8856 (Revision 159566351bc5aa6b83d400c2f34f508cf05f1a8f) Result = SUCCESS kohsuke : 159566351bc5aa6b83d400c2f34f508cf05f1a8f Files : pom.xml changelog.html
            Hide
            jkeller Jurgen Keller added a comment -

            I still encounter this issue with Jenkins 1.521. Client is a PowerMac. In lack of an official PPC JRE it runs an OpenJDK7 build I've got from the Internet (and that might be part of the problem).

            SEVERE: I/O error in channel ch01mac04
            hudson.remoting.DiagnosedStreamCorruptionException
            Read ahead: 0a23204120666174616c206572726f7220686173206265656e20646574656374656420627920746865204a6176612052756e74696d6520456e7669726f6e6d656e743a0a230a232020496e7465726e616c204572726f7220286f735f6273645f7a65726f2e6370703a323332292c207069643d31373438392c207469643d343033333933333331320a2320204572726f723a2063617567687420756e68616e646c6564207369676e616c2031300a230a23204a52452076657273696f6e3a20372e300a23204a61766120564d3a204f70656e4a444b205a65726f20564d202831372e302d62303520696e746572707265746564206d6f6465206273642d70706320290a2320416e206572726f72207265706f72742066696c652077697468206d6f726520696e666f726d6174696f6e2069732073617665642061733a0a23202f55736572732f636f7265746573742f6a656e6b696e732f68735f6572725f70696431373438392e6c6f670a230a2320496620796f7520776f756c64206c696b6520746f207375626d6974206120627567207265706f72742c20706c656173652076697369743a0a23202020687474703a2f2f6a6176612e73756e2e636f6d2f776562617070732f6275677265706f72742f63726173682e6a73700a230a
            	at hudson.remoting.ClassicCommandTransport.diagnoseStreamCorruption(ClassicCommandTransport.java:118)
            	at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:74)
            	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
            Caused by: java.io.StreamCorruptedException: invalid type code: 23
            	at java.io.ObjectInputStream.readObject0(Unknown Source)
            	at java.io.ObjectInputStream.readObject(Unknown Source)
            	at hudson.remoting.Command.readFrom(Command.java:92)
            	at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:72)
            	... 1 more
            
            Show
            jkeller Jurgen Keller added a comment - I still encounter this issue with Jenkins 1.521. Client is a PowerMac. In lack of an official PPC JRE it runs an OpenJDK7 build I've got from the Internet (and that might be part of the problem). SEVERE: I/O error in channel ch01mac04 hudson.remoting.DiagnosedStreamCorruptionException Read ahead: 0a23204120666174616c206572726f7220686173206265656e20646574656374656420627920746865204a6176612052756e74696d6520456e7669726f6e6d656e743a0a230a232020496e7465726e616c204572726f7220286f735f6273645f7a65726f2e6370703a323332292c207069643d31373438392c207469643d343033333933333331320a2320204572726f723a2063617567687420756e68616e646c6564207369676e616c2031300a230a23204a52452076657273696f6e3a20372e300a23204a61766120564d3a204f70656e4a444b205a65726f20564d202831372e302d62303520696e746572707265746564206d6f6465206273642d70706320290a2320416e206572726f72207265706f72742066696c652077697468206d6f726520696e666f726d6174696f6e2069732073617665642061733a0a23202f55736572732f636f7265746573742f6a656e6b696e732f68735f6572725f70696431373438392e6c6f670a230a2320496620796f7520776f756c64206c696b6520746f207375626d6974206120627567207265706f72742c20706c656173652076697369743a0a23202020687474703a2f2f6a6176612e73756e2e636f6d2f776562617070732f6275677265706f72742f63726173682e6a73700a230a at hudson.remoting.ClassicCommandTransport.diagnoseStreamCorruption(ClassicCommandTransport.java:118) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:74) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) Caused by: java.io.StreamCorruptedException: invalid type code: 23 at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:72) ... 1 more
            Hide
            danielbeck Daniel Beck added a comment - - edited

            Decoded read ahead hex string in Jurgen's error message:

            # A fatal error has been detected by the Java Runtime Environment:
            #
            #  Internal Error (os_bsd_zero.cpp:232), pid=17489, tid=4033933312
            #  Error: caught unhandled signal 10
            #
            # JRE version: 7.0
            # Java VM: OpenJDK Zero VM (17.0-b05 interpreted mode bsd-ppc )
            # An error report file with more information is saved as:
            # /Users/coretest/jenkins/hs_err_pid17489.log
            #
            # If you would like to submit a bug report, please visit:
            #   http://java.sun.com/webapps/bugreport/crash.jsp
            #
            

            The referenced file might contain useful information.

            Show
            danielbeck Daniel Beck added a comment - - edited Decoded read ahead hex string in Jurgen's error message: # A fatal error has been detected by the Java Runtime Environment: # # Internal Error (os_bsd_zero.cpp:232), pid=17489, tid=4033933312 # Error: caught unhandled signal 10 # # JRE version: 7.0 # Java VM: OpenJDK Zero VM (17.0-b05 interpreted mode bsd-ppc ) # An error report file with more information is saved as: # /Users/coretest/jenkins/hs_err_pid17489.log # # If you would like to submit a bug report, please visit: # http: //java.sun.com/webapps/bugreport/crash.jsp # The referenced file might contain useful information.
            Hide
            jkeller Jurgen Keller added a comment -

            Thanks for decoding! From the error report I see it crashes in java.lang.String.toCharArray(). Signal 10 might be a memory access/alignment problem. I suppose that's a problem of the JVM I use. If anyone knows of a JVM (1.6 or higher) that works on Mac G5 PowerPC(10.5.8) please let me know. We use that machine for testing our software on a BigEndian platform. Bringing our test environment to another BigEndian platform would be a LOT of work.

            Show
            jkeller Jurgen Keller added a comment - Thanks for decoding! From the error report I see it crashes in java.lang.String.toCharArray(). Signal 10 might be a memory access/alignment problem. I suppose that's a problem of the JVM I use. If anyone knows of a JVM (1.6 or higher) that works on Mac G5 PowerPC(10.5.8) please let me know. We use that machine for testing our software on a BigEndian platform. Bringing our test environment to another BigEndian platform would be a LOT of work.
            danielbeck Daniel Beck made changes -
            Labels lts-candidate
            Hide
            danielbeck Daniel Beck added a comment -

            Nominating the diagnostic read ahead in remoting for LTS.

            Show
            danielbeck Daniel Beck added a comment - Nominating the diagnostic read ahead in remoting for LTS.
            Show
            jglick Jesse Glick added a comment - I created http://repo.jenkins-ci.org/releases/org/jenkins-ci/main/jenkins-war/1.509.2.JENKINS-8856-diag/jenkins-war-1.509.2.JENKINS-8856-diag.war which is 1.509.2 rebuilt with a remoting 2.23 patched with the mentioned https://github.com/jenkinsci/remoting/commit/68be3b1741d8e9e2da8e31491b1b558a8cdbd33b and also https://github.com/jenkinsci/remoting/commit/a468502cf6eb6692fed23884464203d3c2e40305 which seems to be a follow-on.
            Hide
            danielbeck Daniel Beck added a comment -

            Jesse: build appears on Jenkins web site and in lts update center as current best lts version. Just fyi in case this isn't deliberate.

            Show
            danielbeck Daniel Beck added a comment - Jesse: build appears on Jenkins web site and in lts update center as current best lts version. Just fyi in case this isn't deliberate.
            Hide
            stephendonner Stephen Donner added a comment -

            Not only that, but http://mirrors.jenkins-ci.org/war/1.509.2.JENKINS-8856-diag/jenkins.war, which is offered, is a straight-up 404; should I file a separate bug for that?

            Show
            stephendonner Stephen Donner added a comment - Not only that, but http://mirrors.jenkins-ci.org/war/1.509.2.JENKINS-8856-diag/jenkins.war , which is offered, is a straight-up 404; should I file a separate bug for that?
            Hide
            kohsuke Kohsuke Kawaguchi added a comment -

            I updated the update center generator code to exclude these pseudo releases.

            Show
            kohsuke Kohsuke Kawaguchi added a comment - I updated the update center generator code to exclude these pseudo releases.
            Hide
            jglick Jesse Glick added a comment -

            Oops, did not consider that possibility! Thanks Kohsuke for fixing.

            Show
            jglick Jesse Glick added a comment - Oops, did not consider that possibility! Thanks Kohsuke for fixing.
            Show
            ingunawa Indra Gunawan added a comment - Is there a RedHat rpm available for this version? http://repo.jenkins-ci.org/releases/org/jenkins-ci/main/jenkins-war/1.509.2.JENKINS-8856-diag/jenkins-war-1.509.2.JENKINS-8856-diag.war Thanks. -Indra
            Hide
            jglick Jesse Glick added a comment -

            @ingunawa: uh, no. This build is for experimentation only. If you have an RPM installation and want to try it, just overwrite your existing jenkins.war and be prepared to revert.

            Show
            jglick Jesse Glick added a comment - @ingunawa: uh, no. This build is for experimentation only. If you have an RPM installation and want to try it, just overwrite your existing jenkins.war and be prepared to revert.
            Hide
            jglick Jesse Glick added a comment -

            One user reported that the stream corruption was in fact the text Killed, and that /var/log/messages mentioned kernel: java invoked oom-killer, i.e. the slave OS (Linux) had run out of memory and was killing off slave agents. Unfortunately this message seems to have been dumped into the remoting channel and thus confused the master (which saw the K and died). The SSH Slaves plugin is supposed to disconnect the original stdio streams from the channel once the connection is set up, but perhaps this is not working.

            Show
            jglick Jesse Glick added a comment - One user reported that the stream corruption was in fact the text Killed , and that /var/log/messages mentioned kernel: java invoked oom-killer , i.e. the slave OS (Linux) had run out of memory and was killing off slave agents. Unfortunately this message seems to have been dumped into the remoting channel and thus confused the master (which saw the K and died). The SSH Slaves plugin is supposed to disconnect the original stdio streams from the channel once the connection is set up, but perhaps this is not working.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/java/hudson/remoting/HexDump.java
            src/test/java/hudson/remoting/HexDumpTest.java
            http://jenkins-ci.org/commit/remoting/d0334ba25b503038e17488cdbd5dd9bb5cb8121c
            Log:
            JENKINS-8856 Diagnostic improvement: show ‘'i' 'l' 'l' 'e' 'd' 0x0a’ rather than ‘696c6c65640a’, so it is more obviously the rest of ‘Killed’.

            Compare: https://github.com/jenkinsci/remoting/compare/09524af34b57...d0334ba25b50

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/hudson/remoting/HexDump.java src/test/java/hudson/remoting/HexDumpTest.java http://jenkins-ci.org/commit/remoting/d0334ba25b503038e17488cdbd5dd9bb5cb8121c Log: JENKINS-8856 Diagnostic improvement: show ‘'i' 'l' 'l' 'e' 'd' 0x0a’ rather than ‘696c6c65640a’, so it is more obviously the rest of ‘Killed’. Compare: https://github.com/jenkinsci/remoting/compare/09524af34b57...d0334ba25b50
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Daniel Beck
            Path:
            src/main/java/hudson/remoting/HexDump.java
            src/test/java/hudson/remoting/HexDumpTest.java
            http://jenkins-ci.org/commit/remoting/aae9b457c15f88b7a8607978177f1673db2ec2ff
            Log:
            JENKINS-8856 Nicer logging of ASCII text.

            It's no longer a 'hex' dump, but much more readable for all known error cases (JVM dying and writing to standard out).

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: src/main/java/hudson/remoting/HexDump.java src/test/java/hudson/remoting/HexDumpTest.java http://jenkins-ci.org/commit/remoting/aae9b457c15f88b7a8607978177f1673db2ec2ff Log: JENKINS-8856 Nicer logging of ASCII text. It's no longer a 'hex' dump, but much more readable for all known error cases (JVM dying and writing to standard out).
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/java/hudson/remoting/HexDump.java
            src/test/java/hudson/remoting/HexDumpTest.java
            http://jenkins-ci.org/commit/remoting/8b509574e538061ecce717510688bb18acfc3be3
            Log:
            Merge pull request #16 from daniel-beck/master

            More readable diagnostic for JENKINS-8856

            Compare: https://github.com/jenkinsci/remoting/compare/a56730603577...8b509574e538

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/hudson/remoting/HexDump.java src/test/java/hudson/remoting/HexDumpTest.java http://jenkins-ci.org/commit/remoting/8b509574e538061ecce717510688bb18acfc3be3 Log: Merge pull request #16 from daniel-beck/master More readable diagnostic for JENKINS-8856 Compare: https://github.com/jenkinsci/remoting/compare/a56730603577...8b509574e538
            Hide
            danielbeck Daniel Beck added a comment -

            IMO this should be closed. Any new occurrences (1.527+ and future 1.532.x LTS) will be accompanied by much more information and should probably be a new Jira issue about the specific failure indicated by diagnostic output.

            Show
            danielbeck Daniel Beck added a comment - IMO this should be closed. Any new occurrences (1.527+ and future 1.532.x LTS) will be accompanied by much more information and should probably be a new Jira issue about the specific failure indicated by diagnostic output.
            Hide
            jglick Jesse Glick added a comment -

            Agreed, newer reports will probably be more useful.

            Show
            jglick Jesse Glick added a comment - Agreed, newer reports will probably be more useful.
            jglick Jesse Glick made changes -
            Status Reopened [ 4 ] Resolved [ 5 ]
            Resolution Cannot Reproduce [ 5 ]
            danielbeck Daniel Beck made changes -
            Labels lts-candidate
            Show
            jglick Jesse Glick added a comment - New remoting fix in: https://github.com/jenkinsci/jenkins/commit/4137c46182a0652d2eed2897a1e797bdc3f8c7f9
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oliver Gondža
            Path:
            http://jenkins-ci.org/commit/jenkins/5cf3e28c4885a86a4dad3463952e1aae34932f9c
            Log:
            [FIXED JENKINS-20093 JENKINS-19453 JENKINS-19004 JENKINS-8856] Noting retroactively

            Backported as 23424ea7dbced486963a73d6bf56e8367ab779dd

            Compare: https://github.com/jenkinsci/jenkins/compare/7d602d52b445...5cf3e28c4885

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oliver Gondža Path: http://jenkins-ci.org/commit/jenkins/5cf3e28c4885a86a4dad3463952e1aae34932f9c Log: [FIXED JENKINS-20093 JENKINS-19453 JENKINS-19004 JENKINS-8856] Noting retroactively Backported as 23424ea7dbced486963a73d6bf56e8367ab779dd Compare: https://github.com/jenkinsci/jenkins/compare/7d602d52b445...5cf3e28c4885
            olivergondza Oliver Gondža made changes -
            Labels 1.532.2-fixed
            gdm85 Giuseppe Mazzotta made changes -
            Link This issue is related to JENKINS-23110 [ JENKINS-23110 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 139000 ] JNJira + In-Review [ 188280 ]

              People

              • Assignee:
                kohsuke Kohsuke Kawaguchi
                Reporter:
                ebannn i b
              • Votes:
                3 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: