Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46659

TimeoutException from Jetty 9 not recoverable

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      It seems that TimeoutException thrown in Jetty 8, which was recoverable (the channel would stay open and then continue to process data), is fatal in Jetty 9. This will prevent CLI commands for running for more than 30s.

        Attachments

          Issue Links

            Activity

            Hide
            jglick Jesse Glick added a comment -

            CC Olivier Lamy. Discovered by Kohsuke Kawaguchi.

            Show
            jglick Jesse Glick added a comment - CC Olivier Lamy . Discovered by Kohsuke Kawaguchi .
            Hide
            jglick Jesse Glick added a comment -

            Does not seem to be reproducible using java -jar jenkins.war; not sure what might be different in the bundled Jetty. Needs investigation.

            Show
            jglick Jesse Glick added a comment - Does not seem to be reproducible using java -jar jenkins.war ; not sure what might be different in the bundled Jetty. Needs investigation.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            Jenkinsfile
            test/src/test/java/hudson/cli/CLIActionTest.java
            http://jenkins-ci.org/commit/jenkins/810a21534d5183055f303fd6f2bf69eb85e19367
            Log:
            JENKINS-46659 Suppressing failing CLIActionTest.interleavedStdio.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: Jenkinsfile test/src/test/java/hudson/cli/CLIActionTest.java http://jenkins-ci.org/commit/jenkins/810a21534d5183055f303fd6f2bf69eb85e19367 Log: JENKINS-46659 Suppressing failing CLIActionTest.interleavedStdio.
            Hide
            olamy Olivier Lamy added a comment -

            Jesse Glick any stack trace?

            Show
            olamy Olivier Lamy added a comment - Jesse Glick any stack trace?
            Hide
            jglick Jesse Glick added a comment -

            Olivier Lamy

            Sep 08, 2017 5:30:04 PM hudson.cli.PlainCLIProtocol$EitherSide$Reader run
            FINE: ignoring idle timeout, perhaps from Jetty
            java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms
            	at org.eclipse.jetty.server.HttpInput$ErrorState.noContent(HttpInput.java:1047)
            	at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:307)
            	at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:250)
            	at org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:58)
            	at hudson.cli.FlightRecorderInputStream.read(FlightRecorderInputStream.java:88)
            	at java.io.DataInputStream.readInt(DataInputStream.java:387)
            	at hudson.cli.PlainCLIProtocol$EitherSide$Reader.run(PlainCLIProtocol.java:111)
            Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms
            	at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166)
            	at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50)
            	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            	at java.lang.Thread.run(Thread.java:748)
            

            The stack trace on its own is not particularly interesting because we got the same in Jetty 8. The difference is that 8 allowed Jenkins to catch the error but then go right back to reading from the channel, whereas 9 seems to repeatedly throw the same error—it is not recoverable.

            Probably PlainCLIProtocol needs to be extended to include a ping packet type.

            Show
            jglick Jesse Glick added a comment - Olivier Lamy Sep 08, 2017 5:30:04 PM hudson.cli.PlainCLIProtocol$EitherSide$Reader run FINE: ignoring idle timeout, perhaps from Jetty java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms at org.eclipse.jetty.server.HttpInput$ErrorState.noContent(HttpInput.java:1047) at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:307) at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:250) at org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:58) at hudson.cli.FlightRecorderInputStream.read(FlightRecorderInputStream.java:88) at java.io.DataInputStream.readInt(DataInputStream.java:387) at hudson.cli.PlainCLIProtocol$EitherSide$Reader.run(PlainCLIProtocol.java:111) Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166) at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) The stack trace on its own is not particularly interesting because we got the same in Jetty 8. The difference is that 8 allowed Jenkins to catch the error but then go right back to reading from the channel, whereas 9 seems to repeatedly throw the same error—it is not recoverable. Probably PlainCLIProtocol needs to be extended to include a ping packet type.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            cli/src/main/java/hudson/cli/CLI.java
            cli/src/main/java/hudson/cli/PlainCLIProtocol.java
            test/src/test/java/hudson/cli/CLIActionTest.java
            http://jenkins-ci.org/commit/jenkins/dc06c3f03b026c8faafa3e080f3a307c97f4fd42
            Log:
            [FIXED JENKINS-46659] Avoid Jetty timeouts by sending a ping-like packet every 10s while a CLI command is running.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: cli/src/main/java/hudson/cli/CLI.java cli/src/main/java/hudson/cli/PlainCLIProtocol.java test/src/test/java/hudson/cli/CLIActionTest.java http://jenkins-ci.org/commit/jenkins/dc06c3f03b026c8faafa3e080f3a307c97f4fd42 Log: [FIXED JENKINS-46659] Avoid Jetty timeouts by sending a ping-like packet every 10s while a CLI command is running.
            Hide
            danielbeck Daniel Beck added a comment -

            Merged towards 2.80.

            Show
            danielbeck Daniel Beck added a comment - Merged towards 2.80.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            cli/src/main/java/hudson/cli/CLI.java
            cli/src/main/java/hudson/cli/PlainCLIProtocol.java
            http://jenkins-ci.org/commit/jenkins/dda5903c99a46df0d70b7d143d4fd14389bd433b
            Log:
            [FIXED JENKINS-46659] Avoid Jetty timeouts by sending a ping-like packet every 10s while a CLI command is running.

            (cherry picked from commit dc06c3f03b026c8faafa3e080f3a307c97f4fd42)

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: cli/src/main/java/hudson/cli/CLI.java cli/src/main/java/hudson/cli/PlainCLIProtocol.java http://jenkins-ci.org/commit/jenkins/dda5903c99a46df0d70b7d143d4fd14389bd433b Log: [FIXED JENKINS-46659] Avoid Jetty timeouts by sending a ping-like packet every 10s while a CLI command is running. (cherry picked from commit dc06c3f03b026c8faafa3e080f3a307c97f4fd42)

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                jglick Jesse Glick
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: