Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-23232

Jenkins CLI: java.io.StreamCorruptedException: invalid stream header: 0A0A0A0A

    Details

    • Similar Issues:

      Description

      We use the jenkins-cli.jar to attach new slaves and this has been working successfully in v1.534, however when upgrading to v1.554.1, we are unable to do so and a "SEVERE: I/O error in channel Chunked connection" error occurs.

      Tested on linux slaves using java version "1.6.0_35" and "1.7.0_45" and the same error occurs.

      Attached console output to error.log

      Workaround: -Dhudson.diyChunking=false on both slave/agent and master sides

        Attachments

          Issue Links

            Activity

            ashokm Ashok Manji created issue -
            ashokm Ashok Manji made changes -
            Field Original Value New Value
            Link This issue is related to JENKINS-20128 [ JENKINS-20128 ]
            ashokm Ashok Manji made changes -
            Component/s core [ 15593 ]
            Component/s master-slave [ 15489 ]
            Hide
            kohsuke Kohsuke Kawaguchi added a comment -

            My guess is that this is because hudson.diyChunking is still set to true, and between Jetty and Jenkins we end up double-decoding chunked encoding. Needs to verify.

            Show
            kohsuke Kohsuke Kawaguchi added a comment - My guess is that this is because hudson.diyChunking is still set to true, and between Jetty and Jenkins we end up double-decoding chunked encoding. Needs to verify.
            Hide
            ashokm Ashok Manji added a comment -

            Thanks for the update Kohsuke. Looking forward to the resolution

            Show
            ashokm Ashok Manji added a comment - Thanks for the update Kohsuke. Looking forward to the resolution
            Hide
            danielbeck Daniel Beck added a comment -

            Ashok: What's the output of

            System.getProperty('hudson.diyChunking')

            in Manage Jenkins » Script Console?

            Show
            danielbeck Daniel Beck added a comment - Ashok: What's the output of System .getProperty( 'hudson.diyChunking' ) in Manage Jenkins » Script Console?
            Hide
            ashokm Ashok Manji added a comment -

            @Daniel Beck - Result: true

            Show
            ashokm Ashok Manji added a comment - @Daniel Beck - Result: true
            Hide
            peter_kline Peter Kline added a comment - - edited

            Possibly related – JENKINS-23223

            Show
            peter_kline Peter Kline added a comment - - edited Possibly related – JENKINS-23223
            peter_kline Peter Kline made changes -
            Link This issue is related to JENKINS-23223 [ JENKINS-23223 ]
            Hide
            danielbeck Daniel Beck added a comment -

            Ashok: Check your startup scripts (e.g. /etc/sysconfig/jenkins, /etc/init.d/jenkins, ... when using the RPM installer) for any occurrence of

            -Dhudson.diyChunking=true

            and remove it. Restart Jenkins using

            service restart jenkins

            (rather than via e.g. /safeRestart) so it picks up the changes. See whether the issue disappears.

            Show
            danielbeck Daniel Beck added a comment - Ashok: Check your startup scripts (e.g. /etc/sysconfig/jenkins, /etc/init.d/jenkins, ... when using the RPM installer) for any occurrence of -Dhudson.diyChunking= true and remove it. Restart Jenkins using service restart jenkins (rather than via e.g. /safeRestart) so it picks up the changes. See whether the issue disappears.
            Hide
            peter_kline Peter Kline added a comment -

            By setting

            -Dhudson.diyChunking=false

            in /etc/sysconfig/jenkins and restarting I am now able to run remote build calls. Not sure if this fixes the remote slave issue but it certainly solves my issue when running the jenkins-cli.jar problems with Chunking

            Show
            peter_kline Peter Kline added a comment - By setting -Dhudson.diyChunking=false in /etc/sysconfig/jenkins and restarting I am now able to run remote build calls. Not sure if this fixes the remote slave issue but it certainly solves my issue when running the jenkins-cli.jar problems with Chunking
            Hide
            danielbeck Daniel Beck added a comment -

            slave-setup is a plugin, removing that component.

            Show
            danielbeck Daniel Beck added a comment - slave-setup is a plugin, removing that component.
            danielbeck Daniel Beck made changes -
            Component/s slave-setup [ 15929 ]
            Hide
            ashokm Ashok Manji added a comment -

            @Daniel - There were no occurrences of "-Dhudson.diyChunking=true" in my startup scripts, yet this is set as true (even though the default is false)
            I followed Peter's comment above to set this property to 'false' in /etc/sysconfig/jenkins and this has resolved the reported problem.

            Show
            ashokm Ashok Manji added a comment - @Daniel - There were no occurrences of "-Dhudson.diyChunking=true" in my startup scripts, yet this is set as true (even though the default is false) I followed Peter's comment above to set this property to 'false' in /etc/sysconfig/jenkins and this has resolved the reported problem.
            Hide
            geerlingguy Jeff Geerling added a comment -

            In my case (1.574, CentOS 6.5), I was getting the same error, but it seems it was a case of DNS resolution failure with my URL. I had to explicitly map the IP of the server to the server's hostname (as used in the jenkins-cli command) and this error went away. The diyChunking property was false when I checked.

            Show
            geerlingguy Jeff Geerling added a comment - In my case (1.574, CentOS 6.5), I was getting the same error, but it seems it was a case of DNS resolution failure with my URL. I had to explicitly map the IP of the server to the server's hostname (as used in the jenkins-cli command) and this error went away. The diyChunking property was false when I checked.
            Hide
            danielbeck Daniel Beck added a comment -

            Given the recent responses, reducing priority.

            Kohsuke Kawaguchi: Is there still a bug left to be fixed? Or is this a doc issue? Or not a defect at all?

            Show
            danielbeck Daniel Beck added a comment - Given the recent responses, reducing priority. Kohsuke Kawaguchi : Is there still a bug left to be fixed? Or is this a doc issue? Or not a defect at all?
            danielbeck Daniel Beck made changes -
            Priority Blocker [ 1 ] Major [ 3 ]
            Hide
            dserodio Daniel Serodio added a comment -

            I've seen this error message when trying to use jenkins-cli pointing to a Jenkins master where the JNLP port is blocked by a firewall

            Show
            dserodio Daniel Serodio added a comment - I've seen this error message when trying to use jenkins-cli pointing to a Jenkins master where the JNLP port is blocked by a firewall
            Hide
            stevejansen Steve Jansen added a comment - - edited

            Not sure if this helps or not, but I faced a similar issue with this network topology:

            • LAN A (OpenStack cluster with a 192.168.2.0/24 internal network, NAT'd to the rest of the LAN via "floating" public IPs on 10.166.140.0/24)
            • Jenkins master @ 192.168.2.10 or 10.166.140.10
            • Jenkins slave 1 @ 192.168.2.11
            • Jenkins slave 2 @ 192.168.2.12
            • LAN B (3rd party data center, connected to LAN A via site-to-site VPN which NAT to 10.21.26.0/24 on LAN A)
            • Jenkins slave 3 @ 10.21.26.50
            • Jenkins slave 4 @ 10.21.26.51

            The CLI worked as expected when using the master node's private ip address but not the master node's public ip address.

            For example `java -jar jenkins-cli.jar -s http://192.168.2.10:8080/ help` worked, but `java -jar jenkins-cli.jar -s http://10.166.140.10:8080/ help` raised the error "SEVERE: I/O error in channel Chunked connection" on any slave.

            The problem was ultimately the software firewall in the OpenStack cluster that only applied to the floating IPs (e.g., 10.166.140.0/24).

            The resolution was to allowing UDP ingress on ports 1024-65535 and TCP ingress on ports 1024-65535 where the destination was the Jenkins master node and the source was my slaves public IPs (10.21.26.0/24 and 10.166.140.0/24).

            After the firewall change, executing `java -jar jenkins-cli.jar -s http://10.166.140.10:8080/ help` worked as expected on all slaves in both data centers.

            My $0.02 - there is some complex networking for CLI connections: there seems to be ICMP, UDP, and TCP in use at the same time, along with random (JNLP?) ports on the server. The CLI client could use connectivity checks and/or clearcut error messages to better explain when connectivity fails for a specific protocol/address/port.

            Show
            stevejansen Steve Jansen added a comment - - edited Not sure if this helps or not, but I faced a similar issue with this network topology: LAN A (OpenStack cluster with a 192.168.2.0/24 internal network, NAT'd to the rest of the LAN via "floating" public IPs on 10.166.140.0/24) Jenkins master @ 192.168.2.10 or 10.166.140.10 Jenkins slave 1 @ 192.168.2.11 Jenkins slave 2 @ 192.168.2.12 LAN B (3rd party data center, connected to LAN A via site-to-site VPN which NAT to 10.21.26.0/24 on LAN A) Jenkins slave 3 @ 10.21.26.50 Jenkins slave 4 @ 10.21.26.51 The CLI worked as expected when using the master node's private ip address but not the master node's public ip address. For example `java -jar jenkins-cli.jar -s http://192.168.2.10:8080/ help` worked, but `java -jar jenkins-cli.jar -s http://10.166.140.10:8080/ help` raised the error "SEVERE: I/O error in channel Chunked connection" on any slave. The problem was ultimately the software firewall in the OpenStack cluster that only applied to the floating IPs (e.g., 10.166.140.0/24). The resolution was to allowing UDP ingress on ports 1024-65535 and TCP ingress on ports 1024-65535 where the destination was the Jenkins master node and the source was my slaves public IPs (10.21.26.0/24 and 10.166.140.0/24). After the firewall change, executing `java -jar jenkins-cli.jar -s http://10.166.140.10:8080/ help` worked as expected on all slaves in both data centers. My $0.02 - there is some complex networking for CLI connections: there seems to be ICMP, UDP, and TCP in use at the same time, along with random (JNLP?) ports on the server. The CLI client could use connectivity checks and/or clearcut error messages to better explain when connectivity fails for a specific protocol/address/port.
            Hide
            dserodio Daniel Serodio added a comment -

            Thanks Steve, it does help. However, we'd have a hard time asking our security team to open a firewall rule allowing so many ports (1024-65535). I don't understand the details of Jenkins CLI networking, is it possible to narrow the range of ports, or are they this much dynamic?

            Show
            dserodio Daniel Serodio added a comment - Thanks Steve, it does help. However, we'd have a hard time asking our security team to open a firewall rule allowing so many ports (1024-65535). I don't understand the details of Jenkins CLI networking, is it possible to narrow the range of ports, or are they this much dynamic?
            Hide
            danielbeck Daniel Beck added a comment -

            Doesn't it just use the port configured on the Configure Global Security page? (needs security enabled)

            Show
            danielbeck Daniel Beck added a comment - Doesn't it just use the port configured on the Configure Global Security page? (needs security enabled)
            Hide
            stevejansen Steve Jansen added a comment -

            Hi Daniel & Daniel,

            I'd love to use a more specific firewall rule. Re: the Configure Global Security port, I think that port is only for JNLP. I have the default setting to use a random port for JNLP. Does a static port for JNLP also effect the CLI? That probably makes sense for the CLI to use JNLP...

            Cheers,
            Steve

            Show
            stevejansen Steve Jansen added a comment - Hi Daniel & Daniel, I'd love to use a more specific firewall rule. Re: the Configure Global Security port, I think that port is only for JNLP. I have the default setting to use a random port for JNLP. Does a static port for JNLP also effect the CLI? That probably makes sense for the CLI to use JNLP... Cheers, Steve
            Show
            danielbeck Daniel Beck added a comment - https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/Functions.java#L1869 https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/TcpSlaveAgentListener.java#L42
            Hide
            cepnyci cepnyci added a comment - - edited

            We experience the same issue with this network topology:

            Jenkins master @ 10.10.100.204
            Jenkins slave 1 @ 10.10.100.205
            Jenkins slave 2 @ 10.10.100.207
            Jenkins slave 3 @ 10.10.100.193

            on the same LAN.

            Jenkins ver. 1.584

            Oct 16, 2014 12:18:27 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run
            SEVERE: I/O error in channel 10.10.100.207
            java.io.StreamCorruptedException: invalid stream header: 6E6C6F61
                    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:782)
                    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:279)
                    at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:40)
                    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
                    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
            
            09:18:06 FATAL: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid stream header: 6E6C6F61
            09:18:06 hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid stream header: 6E6C6F61
            09:18:06 	at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41)
            09:18:06 	at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34)
            09:18:06 	at hudson.remoting.Request.call(Request.java:174)
            09:18:06 	at hudson.remoting.Channel.call(Channel.java:742)
            09:18:06 	at hudson.FilePath.act(FilePath.java:907)
            09:18:06 	at hudson.FilePath.act(FilePath.java:896)
            09:18:06 	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:910)
            09:18:06 	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:845)
            09:18:06 	at hudson.model.AbstractProject.checkout(AbstractProject.java:1253)
            09:18:06 	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:622)
            09:18:06 	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
            09:18:06 	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:528)
            09:18:06 	at hudson.model.Run.execute(Run.java:1754)
            09:18:06 	at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:529)
            09:18:06 	at hudson.model.ResourceController.execute(ResourceController.java:89)
            09:18:06 	at hudson.model.Executor.run(Executor.java:240)
            09:18:06 Caused by: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid stream header: 6E6C6F61
            09:18:06 	at hudson.remoting.Request.abort(Request.java:299)
            09:18:06 	at hudson.remoting.Channel.terminate(Channel.java:805)
            09:18:06 	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
            09:18:06 Caused by: java.io.StreamCorruptedException: invalid stream header: 6E6C6F61
            09:18:06 	at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:782)
            09:18:06 	at java.io.ObjectInputStream.<init>(ObjectInputStream.java:279)
            09:18:06 	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:40)
            09:18:06 	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
            09:18:06 	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
            
            Show
            cepnyci cepnyci added a comment - - edited We experience the same issue with this network topology: Jenkins master @ 10.10.100.204 Jenkins slave 1 @ 10.10.100.205 Jenkins slave 2 @ 10.10.100.207 Jenkins slave 3 @ 10.10.100.193 on the same LAN. Jenkins ver. 1.584 Oct 16, 2014 12:18:27 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run SEVERE: I/O error in channel 10.10.100.207 java.io.StreamCorruptedException: invalid stream header: 6E6C6F61 at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:782) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:279) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:40) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) 09:18:06 FATAL: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid stream header: 6E6C6F61 09:18:06 hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid stream header: 6E6C6F61 09:18:06 at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41) 09:18:06 at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34) 09:18:06 at hudson.remoting.Request.call(Request.java:174) 09:18:06 at hudson.remoting.Channel.call(Channel.java:742) 09:18:06 at hudson.FilePath.act(FilePath.java:907) 09:18:06 at hudson.FilePath.act(FilePath.java:896) 09:18:06 at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:910) 09:18:06 at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:845) 09:18:06 at hudson.model.AbstractProject.checkout(AbstractProject.java:1253) 09:18:06 at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:622) 09:18:06 at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) 09:18:06 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:528) 09:18:06 at hudson.model.Run.execute(Run.java:1754) 09:18:06 at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:529) 09:18:06 at hudson.model.ResourceController.execute(ResourceController.java:89) 09:18:06 at hudson.model.Executor.run(Executor.java:240) 09:18:06 Caused by: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid stream header: 6E6C6F61 09:18:06 at hudson.remoting.Request.abort(Request.java:299) 09:18:06 at hudson.remoting.Channel.terminate(Channel.java:805) 09:18:06 at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69) 09:18:06 Caused by: java.io.StreamCorruptedException: invalid stream header: 6E6C6F61 09:18:06 at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:782) 09:18:06 at java.io.ObjectInputStream.<init>(ObjectInputStream.java:279) 09:18:06 at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:40) 09:18:06 at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) 09:18:06 at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
            Hide
            terrapin Jan Hollevoet added a comment -

            We are at v1.559 and intensively use the dist-fork plugin. This error hit us severely. Since a fixed port is used for JNLP slave agents (Global Security), the error did not occur anymore.

            Show
            terrapin Jan Hollevoet added a comment - We are at v1.559 and intensively use the dist-fork plugin. This error hit us severely. Since a fixed port is used for JNLP slave agents (Global Security), the error did not occur anymore.
            Hide
            danielbeck Daniel Beck added a comment -

            cepnyci: The stream header is in hex digits and equals ASCII 'nloa'. Does that tell you anything?

            Jan Hollevoet: What did you use before? Disabled or dynamic port?

            Show
            danielbeck Daniel Beck added a comment - cepnyci : The stream header is in hex digits and equals ASCII 'nloa'. Does that tell you anything? Jan Hollevoet : What did you use before? Disabled or dynamic port?
            Hide
            terrapin Jan Hollevoet added a comment - - edited

            Daniel, I was using the dynamic port setting.

            Yesterday, I saw it once again on 480 launches of the cli:

            Nov 7, 2014 7:06:12 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run
            SEVERE: I/O error in channel Chunked connection to http://192.168.100.61:8080/cli
            java.io.IOException: Unexpected termination of the channel

            I am also using the ip address instead of the hostname for the Jenkins server. Not sure whether that helped.

            And hudson.diyChunking = false

            Show
            terrapin Jan Hollevoet added a comment - - edited Daniel, I was using the dynamic port setting. Yesterday, I saw it once again on 480 launches of the cli: Nov 7, 2014 7:06:12 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run SEVERE: I/O error in channel Chunked connection to http://192.168.100.61:8080/cli java.io.IOException: Unexpected termination of the channel I am also using the ip address instead of the hostname for the Jenkins server. Not sure whether that helped. And hudson.diyChunking = false
            Hide
            diegows Diego Woitasen added a comment -

            Using FQDN fails and using 127.0.0.1 works for me too.

            Same error the others reported.

            Show
            diegows Diego Woitasen added a comment - Using FQDN fails and using 127.0.0.1 works for me too. Same error the others reported.
            Hide
            cepnyci cepnyci added a comment - - edited

            Daniel, I think you meant ASCII "nload": unknown host. We will update the configuration accordingly and monitor the slaves.Thanks for the help.

            Show
            cepnyci cepnyci added a comment - - edited Daniel, I think you meant ASCII "nload": unknown host. We will update the configuration accordingly and monitor the slaves.Thanks for the help.
            Hide
            danielbeck Daniel Beck added a comment -

            I got it straight from the error message you provided:

            Caused by: java.io.StreamCorruptedException: invalid stream header: 6E6C6F61

            That hex part translates to 'nloa'. Unfortunately it only ever provides four characters.

            Show
            danielbeck Daniel Beck added a comment - I got it straight from the error message you provided: Caused by: java.io.StreamCorruptedException: invalid stream header: 6E6C6F61 That hex part translates to 'nloa'. Unfortunately it only ever provides four characters.
            Hide
            pwong_mapr Patrick Wong added a comment - - edited

            I get the same issue with the following steps:

            • Client machine is CentOS 6.5 with Java 1.7.0-b71 (from CentOS package java-1.7.0-openjdk-devel)
            • Jenkins VM is CentOS 7 with Java 1.7.0-b71 (from CentOS package java-1.7.0-openjdk-devel)
              • port 8080 forwarded to host machine port 8081
              • Vagrant basebox is jayunit/centos7 with Oracle Virtualbox provider
            • jenkins-cli.jar comes from wget localhost:8081/jnlpjars/jenkins-cli.jar and it is then renamed to have the version string, e.g. mv jenkins-cli.jar jenkins-cli-1.580.2.jar
            • Command is java -jar jenkins-cli-1.580.2.jar -s http://localhost:8081/ list-plugins

            Have tried with both Jenkins 1.580.2 stable and 1.596 latest.

            Show
            pwong_mapr Patrick Wong added a comment - - edited I get the same issue with the following steps: Client machine is CentOS 6.5 with Java 1.7.0-b71 (from CentOS package java-1.7.0-openjdk-devel) Jenkins VM is CentOS 7 with Java 1.7.0-b71 (from CentOS package java-1.7.0-openjdk-devel) port 8080 forwarded to host machine port 8081 Vagrant basebox is jayunit/centos7 with Oracle Virtualbox provider jenkins-cli.jar comes from wget localhost:8081/jnlpjars/jenkins-cli.jar and it is then renamed to have the version string, e.g. mv jenkins-cli.jar jenkins-cli-1.580.2.jar Command is java -jar jenkins-cli-1.580.2.jar -s http://localhost:8081/ list-plugins Have tried with both Jenkins 1.580.2 stable and 1.596 latest.
            Hide
            wilfredh Wilfred Hughes added a comment - - edited

            I've seen the same exception with a different header:

            Caused by: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid stream header: 31352D30

            It turned out the cause was my Jenkins slave writing to the channel. My slave configuration was 'Launch slave via execution of the command on the master`, and my script set some JVM flags. The offending flags were:

            -XX:+PrintGCDetails -Xloggc:/foo/bar/jenkins_gc.log

            However, since the /foo/bar directory didn't exist, the JVM wrote GC logs to stdout. This resulted in the Stream seeing garbage. If you decode 31352D30 you get '15-0'. This is the start of a log line of the form:

            2015-01-23T14:01:04.789+0000: 26.069: [CMS-concurrent-sweep-start]

            My workaround was to change my slave to launch via SSH, but I fixed it by ensuring that my GC log directory exists.

            Show
            wilfredh Wilfred Hughes added a comment - - edited I've seen the same exception with a different header: Caused by: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid stream header: 31352D30 It turned out the cause was my Jenkins slave writing to the channel. My slave configuration was 'Launch slave via execution of the command on the master`, and my script set some JVM flags. The offending flags were: -XX:+PrintGCDetails -Xloggc:/foo/bar/jenkins_gc.log However, since the /foo/bar directory didn't exist, the JVM wrote GC logs to stdout. This resulted in the Stream seeing garbage. If you decode 31352D30 you get '15-0'. This is the start of a log line of the form: 2015-01-23T14:01:04.789+0000: 26.069: [CMS-concurrent-sweep-start] My workaround was to change my slave to launch via SSH, but I fixed it by ensuring that my GC log directory exists.
            Hide
            cepnyci cepnyci added a comment -

            We configured the slaves to connect via JNLP. No more experiencing this issue.

            Show
            cepnyci cepnyci added a comment - We configured the slaves to connect via JNLP. No more experiencing this issue.
            Hide
            terrapin Jan Hollevoet added a comment -

            Now I am at 1.565.3. Right after the upgrade, this error was still hitting us. Then I reconfigured all Windows slaves as JNLP nodes with the same revision of the client software (2.46). After this, the situation improved a lot. The number /usr/sbin/lsof | grep protocol | grep [jenkins-pid] | wc -l increased less fast than before. And the server is much more stable.

            I have the impression that the Windows clients are the root cause of the error. Maybe I should reconfigure them simply as SSH slaves.

            Show
            terrapin Jan Hollevoet added a comment - Now I am at 1.565.3. Right after the upgrade, this error was still hitting us. Then I reconfigured all Windows slaves as JNLP nodes with the same revision of the client software (2.46). After this, the situation improved a lot. The number /usr/sbin/lsof | grep protocol | grep [jenkins-pid] | wc -l increased less fast than before. And the server is much more stable. I have the impression that the Windows clients are the root cause of the error. Maybe I should reconfigure them simply as SSH slaves.
            Hide
            ldachary Loic Dachary added a comment -

            This error shows up when the JNLP port (50000 typically) is blocked. It would be very helpful to display an error such as : unable to connect to port XXXX or something instead.

            SEVERE: I/O error in channel Chunked connection to http://127.0.0.1:11080/cli

            Show
            ldachary Loic Dachary added a comment - This error shows up when the JNLP port (50000 typically) is blocked. It would be very helpful to display an error such as : unable to connect to port XXXX or something instead. SEVERE: I/O error in channel Chunked connection to http://127.0.0.1:11080/cli
            Hide
            pere Pere Orga added a comment -

            I got rid of this by enabling the TCP port for JNLP slave agents

            Thank you.

            Show
            pere Pere Orga added a comment - I got rid of this by enabling the TCP port for JNLP slave agents Thank you.
            Hide
            slawomir_czarko Slawomir Czarko added a comment -

            Still happens randomly with 1.625.1 LTS when using jenkins-cli.jar to upload job config.xml file. Sometimes it fails after couple of requests sometimes after 20+.
            Server and client are running on CentOS 7.1
            I have -Dhudson.diyChunking=false in /etc/sysconfig/jenkins
            Enabling TCP port for JNLP slave agents seems to fix this or at least reduce the frequency.

            Show
            slawomir_czarko Slawomir Czarko added a comment - Still happens randomly with 1.625.1 LTS when using jenkins-cli.jar to upload job config.xml file. Sometimes it fails after couple of requests sometimes after 20+. Server and client are running on CentOS 7.1 I have -Dhudson.diyChunking=false in /etc/sysconfig/jenkins Enabling TCP port for JNLP slave agents seems to fix this or at least reduce the frequency.
            Hide
            cygoonda Jeff Feldstein added a comment - - edited

            I've been working with this all day today in both 1.6.25.2 and 1.638
            Centos 6.5
            jenkins-build-monitor plugin is installed
            If the default view is "All" in config.xml, jenkins-cli works.
            If the default view is changed to a view defined by jenkins-build-monitor, the following stack trace occurs:

            oot@zmon10 jenkins]# java -jar /usr/local/bin/jenkins-cli.jar -s http://localhost:8080 version
            Nov 13, 2015 1:50:10 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run
            SEVERE: I/O error in channel Chunked connection to http://localhost:8080/cli
            java.io.StreamCorruptedException: invalid stream header: 0A0A0A0A
            	at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:806)
            	at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
            	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
            	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
            	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
            
            hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid stream header: 0A0A0A0A
            	at hudson.remoting.Request.abort(Request.java:297)
            	at hudson.remoting.Channel.terminate(Channel.java:847)
            	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
            	at ......remote call to Chunked connection to http://localhost:8080/cli(Native Method)
            	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
            	at hudson.remoting.Request.call(Request.java:172)
            	at hudson.remoting.Channel.call(Channel.java:780)
            	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:250)
            	at hudson.remoting.$Proxy1.waitForProperty(Unknown Source)
            	at hudson.remoting.Channel.waitForRemoteProperty(Channel.java:1258)
            	at hudson.cli.CLI.<init>(CLI.java:147)
            	at hudson.cli.CLIConnectionFactory.connect(CLIConnectionFactory.java:72)
            	at hudson.cli.CLI._main(CLI.java:479)
            	at hudson.cli.CLI.main(CLI.java:390)
            Caused by: java.io.StreamCorruptedException: invalid stream header: 0A0A0A0A
            	at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:806)
            	at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
            	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
            	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
            	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
            

            This line in config.xml works:

             <primaryView>All</primaryView>
            

            This line in config.xml causes the stack trace above:

             <primaryView>SYNCl</primaryView>
            
            Show
            cygoonda Jeff Feldstein added a comment - - edited I've been working with this all day today in both 1.6.25.2 and 1.638 Centos 6.5 jenkins-build-monitor plugin is installed If the default view is "All" in config.xml, jenkins-cli works. If the default view is changed to a view defined by jenkins-build-monitor, the following stack trace occurs: oot@zmon10 jenkins]# java -jar /usr/local/bin/jenkins-cli.jar -s http: //localhost:8080 version Nov 13, 2015 1:50:10 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run SEVERE: I/O error in channel Chunked connection to http: //localhost:8080/cli java.io.StreamCorruptedException: invalid stream header: 0A0A0A0A at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:806) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid stream header: 0A0A0A0A at hudson.remoting.Request.abort(Request.java:297) at hudson.remoting.Channel.terminate(Channel.java:847) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69) at ......remote call to Chunked connection to http: //localhost:8080/cli(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416) at hudson.remoting.Request.call(Request.java:172) at hudson.remoting.Channel.call(Channel.java:780) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:250) at hudson.remoting.$Proxy1.waitForProperty(Unknown Source) at hudson.remoting.Channel.waitForRemoteProperty(Channel.java:1258) at hudson.cli.CLI.<init>(CLI.java:147) at hudson.cli.CLIConnectionFactory.connect(CLIConnectionFactory.java:72) at hudson.cli.CLI._main(CLI.java:479) at hudson.cli.CLI.main(CLI.java:390) Caused by: java.io.StreamCorruptedException: invalid stream header: 0A0A0A0A at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:806) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) This line in config.xml works: <primaryView>All</primaryView> This line in config.xml causes the stack trace above: <primaryView>SYNCl</primaryView>
            danielbeck Daniel Beck made changes -
            Link This issue is duplicated by JENKINS-34287 [ JENKINS-34287 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 155793 ] JNJira + In-Review [ 179125 ]
            oleg_nenashev Oleg Nenashev made changes -
            Summary SEVERE: I/O error in channel Chunked connection Jenkins CLI: java.io.StreamCorruptedException: invalid stream header: 0A0A0A0A
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue is duplicated by JENKINS-35197 [ JENKINS-35197 ]
            oleg_nenashev Oleg Nenashev made changes -
            Description We use the jenkins-cli.jar to attach new slaves and this has been working successfully in v1.534, however when upgrading to v1.554.1, we are unable to do so and a "SEVERE: I/O error in channel Chunked connection" error occurs.

            Tested on linux slaves using java version "1.6.0_35" and "1.7.0_45" and the same error occurs.

            Attached console output to error.log
            We use the jenkins-cli.jar to attach new slaves and this has been working successfully in v1.534, however when upgrading to v1.554.1, we are unable to do so and a "SEVERE: I/O error in channel Chunked connection" error occurs.

            Tested on linux slaves using java version "1.6.0_35" and "1.7.0_45" and the same error occurs.

            Attached console output to error.log

            Workaround: -Dhudson.diyChunking=false
            oleg_nenashev Oleg Nenashev made changes -
            Description We use the jenkins-cli.jar to attach new slaves and this has been working successfully in v1.534, however when upgrading to v1.554.1, we are unable to do so and a "SEVERE: I/O error in channel Chunked connection" error occurs.

            Tested on linux slaves using java version "1.6.0_35" and "1.7.0_45" and the same error occurs.

            Attached console output to error.log

            Workaround: -Dhudson.diyChunking=false
            We use the jenkins-cli.jar to attach new slaves and this has been working successfully in v1.534, however when upgrading to v1.554.1, we are unable to do so and a "SEVERE: I/O error in channel Chunked connection" error occurs.

            Tested on linux slaves using java version "1.6.0_35" and "1.7.0_45" and the same error occurs.

            Attached console output to error.log

            Workaround: -Dhudson.diyChunking=false on both slave/agent and master sides
            Hide
            danielbeck Daniel Beck added a comment -

            Needs a release of executable-war with https://github.com/jenkinsci/extras-executable-war/commit/5a5f01362e7649da3b39a3a45940ba3ce1ab65e8

            Oleg Nenashev Any idea whether you can release this?

            Show
            danielbeck Daniel Beck added a comment - Needs a release of executable-war with https://github.com/jenkinsci/extras-executable-war/commit/5a5f01362e7649da3b39a3a45940ba3ce1ab65e8 Oleg Nenashev Any idea whether you can release this?
            oleg_nenashev Oleg Nenashev made changes -
            Assignee Kohsuke Kawaguchi [ kohsuke ] Oleg Nenashev [ oleg_nenashev ]
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Daniel Beck Not sure, but I'll give a try. FTR JENKINS-34287

            Show
            oleg_nenashev Oleg Nenashev added a comment - Daniel Beck Not sure, but I'll give a try. FTR JENKINS-34287
            Hide
            oleg_nenashev Oleg Nenashev added a comment -
            Show
            oleg_nenashev Oleg Nenashev added a comment - PR is ready for review: https://github.com/jenkinsci/extras-executable-war/pull/7
            oleg_nenashev Oleg Nenashev made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            pom.xml
            src/main/java/Main.java
            http://jenkins-ci.org/commit/extras-executable-war/b12662aed40ee06a75c03a51a6fc87a75c642179
            Log:
            [JENKINS-34287,JENKINS-23232] Update the component towards 1.34 (#7)

            • Update to the latest Parent POM and Java 7
            • FindBugs: Propagate issues with file deletion
            • Pick parent POM 1.37 with Java 7 support
            • FindBugs: Fix resource leak when exception happens during the dependency list read
            • Downgrade the code to Java 6
            • Downgrade back to Java 1.3
            • Address comment from @rsandell. Just in case
            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: pom.xml src/main/java/Main.java http://jenkins-ci.org/commit/extras-executable-war/b12662aed40ee06a75c03a51a6fc87a75c642179 Log: [JENKINS-34287,JENKINS-23232] Update the component towards 1.34 (#7) Update to the latest Parent POM and Java 7 FindBugs: Propagate issues with file deletion Pick parent POM 1.37 with Java 7 support FindBugs: Fix resource leak when exception happens during the dependency list read Downgrade the code to Java 6 Downgrade back to Java 1.3 Address comment from @rsandell. Just in case
            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: war/pom.xml http://jenkins-ci.org/commit/jenkins/24443acb3c92612dc623849c9458000c98a0a265 Log: [JENKINS-34287,JENKINS-23232] Update executable-war to 1.34 Diff: https://github.com/jenkinsci/extras-executable-war/compare/executable-war-1.33...executable-war-1.34 https://issues.jenkins-ci.org/browse/JENKINS-23232 https://issues.jenkins-ci.org/browse/JENKINS-34287
            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: war/pom.xml http://jenkins-ci.org/commit/jenkins/d7598f118fd6d9989164cd2ff7126001a6ee2ff2 Log: [JENKINS-34287,JENKINS-23232] Update executable-war to 1.34 (#2565) Diff: https://github.com/jenkinsci/extras-executable-war/compare/executable-war-1.33...executable-war-1.34 https://issues.jenkins-ci.org/browse/JENKINS-23232 https://issues.jenkins-ci.org/browse/JENKINS-34287
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            The fix is integrated towards 2.24

            Show
            oleg_nenashev Oleg Nenashev added a comment - The fix is integrated towards 2.24
            oleg_nenashev Oleg Nenashev made changes -
            Status In Progress [ 3 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            oleg_nenashev Oleg Nenashev made changes -
            Labels 1.554.1 cli exception jenkins severe 1.554.1 cli exception jenkins lts-candidate severe
            olivergondza Oliver Gondža made changes -
            Labels 1.554.1 cli exception jenkins lts-candidate severe 1.554.1 2.19.2-fixed cli exception jenkins severe

              People

              • Assignee:
                oleg_nenashev Oleg Nenashev
                Reporter:
                ashokm Ashok Manji
              • Votes:
                6 Vote for this issue
                Watchers:
                22 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: