Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-12037

CLI - I/O error in channel Chunked connection/Unexpected termination of the channel - still occurring in Jenkins 1.449

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: cli
    • Labels:
      None
    • Environment:
      * Running on SLES9 Linux server with 4 CPUs and plenty of diskspace.
      * Tomcat 7.0.22
      * JDK 1.6.0_14
      * Only ONE Master configuration - no slaves are configured
      * 3 Executors - (one less than the max number of CPUs)

      Description

      We reported an issue some time back that was also listed as fixed in Jenkins 1.441:
      Log:
      [FIXED JENKINS-11130] SEVERE: I/O error in channel Chunked connection when using jenkins-cli.jar

      Perhaps another bug should NOT be submitted so I have added the following comments below the line to the original defect 11130 comments section in case it can be reviewed/re-opened.

      We did NOT try to make any adjustments to the Tomcat configuration:

      Tomcat Connector connectionUploadTimeout

      but we are also now seeing the same problem with Winstone when at this same 1.441 level. We did revert to the 1.438 version of the CLI (leaving the WAR at 1.441 running in Winstone) and that is serving asthe current workaround.

      ================================================================================================

      We have downloaded and installed the LATEST 1.441 release that lists the fix for this problem. Currently we were running 1.438 on Winstone only (since with Tomcat 6 or 7, we had experienced the error HOWEVER yet under Winstone, it worked OK so that was our workaround - while running 1.438).

      Now with Jenkins 1.441 - we are getting the ERROR again and NOW WITH BOTH Winstone and the Tomcat configurations). We have left the Jenkins 1.441 WAR file in place running on Winstone, and reverted the CLI jar file back to the 1.438 version for now and that appears to work again with Winstone.

      Checked Manifest of CLI jar downloaded with the 1.441 WAR installation:

      Manifest-Version: 1.0
      Archiver-Version: Plexus Archiver
      Created-By: Apache Maven
      Built-By: kohsuke
      Build-Jdk: 1.6.0_26
      Main-Class: hudson.cli.CLI
      Jenkins-CLI-Version: 1.441

      Under Tomcat 7, we get this stacktrace:

      Started by command line
      [workspace] $ /bin/bash -xe /opt/apache-tomcat-7.0.22_jenkins/temp/hudson32817888834817830.sh
      + /opt/Sun/jdk1.6.0_14/bin/java -jar /opt/Sun/jdk1.6.0_14/lib/jenkins-cli.jar -s http://11.22.33.44:8082/jenkins/ build XYZ_Project-SharedLibs -s -p SVN_PATH=trunk
      Dec 5, 2011 12:59:11 PM hudson.remoting.Channel$ReaderThread run
      SEVERE: I/O error in channel Chunked connection to http://11.22.33.44:8082/jenkins/cli
      java.io.IOException: Unexpected termination of the channel
      at hudson.remoting.Channel$ReaderThread.run(Channel.java:1115)
      Caused by: java.io.EOFException
      at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
      at hudson.remoting.Channel$ReaderThread.run(Channel.java:1109)
      Exception in thread "main" hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
      at hudson.remoting.Request.call(Request.java:149)
      at hudson.remoting.Channel.call(Channel.java:681)
      at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:158)
      at $Proxy2.main(Unknown Source)
      at hudson.cli.CLI.execute(CLI.java:200)
      at hudson.cli.CLI._main(CLI.java:330)
      at hudson.cli.CLI.main(CLI.java:245)
      Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
      at hudson.remoting.Request.abort(Request.java:273)
      at hudson.remoting.Channel.terminate(Channel.java:732)
      at hudson.remoting.Channel$ReaderThread.run(Channel.java:1139)
      Caused by: java.io.IOException: Unexpected termination of the channel
      at hudson.remoting.Channel$ReaderThread.run(Channel.java:1115)
      Caused by: java.io.EOFException
      at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
      at hudson.remoting.Channel$ReaderThread.run(Channel.java:1109)
      Build step 'Execute shell' marked build as failure
      Notifying upstream projects of job completion
      Finished: FAILURE

      Under Winstone, we get this stacktrace - it's somewhat different:

      Started by command line
      [workspace] $ /bin/bash -xe /tmp/hudson10791816374444704.sh
      + /opt/Sun/jdk1.6.0_14/bin/java -jar /opt/Sun/jdk1.6.0_14/lib/jenkins-cli.jar -s http://11.22.33.44:8082/jenkins/ build XYZ_Project-SharedLibs -s -p SVN_PATH=trunk
      Dec 5, 2011 1:18:22 PM hudson.remoting.Channel$ReaderThread run
      SEVERE: I/O error in channel Chunked connection to http://11.22.33.44:8082/jenkins/cli
      java.io.IOException: Premature EOF
      at sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:538) (http://www.http.ChunkedInputStream.readAheadBlocking%28ChunkedInputStream.java:538%29)
      at sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:582) (http://www.http.ChunkedInputStream.readAhead%28ChunkedInputStream.java:582%29)
      at sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:669) (http://www.http.ChunkedInputStream.read%28ChunkedInputStream.java:669%29)
      at java.io.FilterInputStream.read(FilterInputStream.java:116)
      at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2504) (http://www.protocol.http.HttpURLConnection$HttpInputStream.read%28HttpURLConnection.java:2504%29)
      at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2499) (http://www.protocol.http.HttpURLConnection$HttpInputStream.read%28HttpURLConnection.java:2499%29)
      at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2488) (http://www.protocol.http.HttpURLConnection$HttpInputStream.read%28HttpURLConnection.java:2488%29)
      at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2249)
      at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2542)
      at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2552)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
      at hudson.remoting.Channel$ReaderThread.run(Channel.java:1109)
      Exception in thread "main" hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Premature EOF
      at hudson.remoting.Request.call(Request.java:149)
      at hudson.remoting.Channel.call(Channel.java:681)
      at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:158)
      at $Proxy2.main(Unknown Source)
      at hudson.cli.CLI.execute(CLI.java:200)
      at hudson.cli.CLI._main(CLI.java:330)
      at hudson.cli.CLI.main(CLI.java:245)
      Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Premature EOF
      at hudson.remoting.Request.abort(Request.java:273)
      at hudson.remoting.Channel.terminate(Channel.java:732)
      at hudson.remoting.Channel$ReaderThread.run(Channel.java:1139)
      Caused by: java.io.IOException: Premature EOF
      at sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:538) (http://www.http.ChunkedInputStream.readAheadBlocking%28ChunkedInputStream.java:538%29)
      at sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:582) (http://www.http.ChunkedInputStream.readAhead%28ChunkedInputStream.java:582%29)
      at sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:669) (http://www.http.ChunkedInputStream.read%28ChunkedInputStream.java:669%29)
      at java.io.FilterInputStream.read(FilterInputStream.java:116)
      at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2504) (http://www.protocol.http.HttpURLConnection$HttpInputStream.read%28HttpURLConnection.java:2504%29)
      at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2499) (http://www.protocol.http.HttpURLConnection$HttpInputStream.read%28HttpURLConnection.java:2499%29)
      at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2488) (http://www.protocol.http.HttpURLConnection$HttpInputStream.read%28HttpURLConnection.java:2488%29)
      at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2249)
      at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2542)
      at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2552)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
      at hudson.remoting.Channel$ReaderThread.run(Channel.java:1109)
      Build step 'Execute shell' marked build as failure
      Notifying upstream projects of job completion
      Finished: FAILURE

      1. jenkins-timeout
        39 kB
        George Panayotov

        Activity

        Hide
        oldelvet Richard Mortimer added a comment -

        Yes it will be part of 1.459. The snapshot build that you tested is direct from Jenkins on Jenkins builds.

        I mentioned it earlier but the current fix actually busy-waits during the ping cycle so you may see high cpu usage from the cli command. The busy-wait is fixed by the change that I have submitted to the underlying "remoting" library. That might take a release or two to filter through.

        Finally we should really say thank you to George who managed to get a packet trace with the failure so that we could work out what was going on.

        Show
        oldelvet Richard Mortimer added a comment - Yes it will be part of 1.459. The snapshot build that you tested is direct from Jenkins on Jenkins builds. I mentioned it earlier but the current fix actually busy-waits during the ping cycle so you may see high cpu usage from the cli command. The busy-wait is fixed by the change that I have submitted to the underlying "remoting" library. That might take a release or two to filter through. Finally we should really say thank you to George who managed to get a packet trace with the failure so that we could work out what was going on.
        Hide
        dogfood dogfood added a comment -

        Integrated in jenkins_ui-changes_branch #21
        [FIXED JENKINS-12037] CLI - I/O error in channel Chunked connection (Revision ea1b80aebe85a9414d5befd58e976d85818a258d)

        Result = SUCCESS
        richm : ea1b80aebe85a9414d5befd58e976d85818a258d
        Files :

        • cli/src/main/java/hudson/cli/CLI.java
        • changelog.html
        Show
        dogfood dogfood added a comment - Integrated in jenkins_ui-changes_branch #21 [FIXED JENKINS-12037] CLI - I/O error in channel Chunked connection (Revision ea1b80aebe85a9414d5befd58e976d85818a258d) Result = SUCCESS richm : ea1b80aebe85a9414d5befd58e976d85818a258d Files : cli/src/main/java/hudson/cli/CLI.java changelog.html
        Hide
        moshe_sucaz Moshe Sucaz added a comment -

        Hi,
        We have just downloaded Jenkins 1.461 (because we had this problem with 1.443), and I still see such failures in catalina logs. we are using tomcat 6.0.29. will upgrade to Tomcat 7.0.22 help?

        Show
        moshe_sucaz Moshe Sucaz added a comment - Hi, We have just downloaded Jenkins 1.461 (because we had this problem with 1.443), and I still see such failures in catalina logs. we are using tomcat 6.0.29. will upgrade to Tomcat 7.0.22 help?
        Hide
        mcs13099 mark streit added a comment -

        A couple of things with this:

        1) you need to make sure you download and install the latest jenkins-cli.jar that comes with 1.460 or higher version of the WAR download...and have it in your classpath wherever you are utilizing the CLI jar.
        2) It's the CLI jar code that was fixed as part of this. Simply updating the WAR file on your Tomcat instance is only part of the upgrade.
        3) I checked our catalina.out logs on our Tomcat 7.0.22 installation and we don't see these anymore since the fix was delivered in 1.460 which is what we now run. We made sure wherever we use CLI commands that the 1.460 version of the jenkins-cli.jar is on the classpath.

        Hope this helps.

        Show
        mcs13099 mark streit added a comment - A couple of things with this: 1) you need to make sure you download and install the latest jenkins-cli.jar that comes with 1.460 or higher version of the WAR download...and have it in your classpath wherever you are utilizing the CLI jar. 2) It's the CLI jar code that was fixed as part of this. Simply updating the WAR file on your Tomcat instance is only part of the upgrade. 3) I checked our catalina.out logs on our Tomcat 7.0.22 installation and we don't see these anymore since the fix was delivered in 1.460 which is what we now run. We made sure wherever we use CLI commands that the 1.460 version of the jenkins-cli.jar is on the classpath. Hope this helps.
        Hide
        scm_issue_link SCM/JIRA link daemon added a comment -

        Code changed in jenkins
        User: Richard Mortimer
        Path:
        src/main/java/hudson/remoting/PingThread.java
        http://jenkins-ci.org/commit/remoting/463b5427d272f5d0c2c97eb6c41ef95757eb6c15
        Log:
        Do not attempt to retry a channel get after an ExecutionException.
        Without this the pinger busy waits retrying the same operation for the
        timeout period. The timeout/retry mechanism was only intended to retry
        pings that were interrupted.

        This is part of the failure seen in JENKINS-12037 where the PingThread
        busy waits for 4 minutes when the Ping class fails to execute on a restricted
        channel.

        Show
        scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Richard Mortimer Path: src/main/java/hudson/remoting/PingThread.java http://jenkins-ci.org/commit/remoting/463b5427d272f5d0c2c97eb6c41ef95757eb6c15 Log: Do not attempt to retry a channel get after an ExecutionException. Without this the pinger busy waits retrying the same operation for the timeout period. The timeout/retry mechanism was only intended to retry pings that were interrupted. This is part of the failure seen in JENKINS-12037 where the PingThread busy waits for 4 minutes when the Ping class fails to execute on a restricted channel.

          People

          • Assignee:
            Unassigned
            Reporter:
            mcs13099 mark streit
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: