Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46659

TimeoutException from Jetty 9 not recoverable

    Details

    • Similar Issues:

      Description

      It seems that TimeoutException thrown in Jetty 8, which was recoverable (the channel would stay open and then continue to process data), is fatal in Jetty 9. This will prevent CLI commands for running for more than 30s.

        Attachments

          Issue Links

            Activity

            jglick Jesse Glick created issue -
            Hide
            jglick Jesse Glick added a comment -

            CC Olivier Lamy. Discovered by Kohsuke Kawaguchi.

            Show
            jglick Jesse Glick added a comment - CC Olivier Lamy . Discovered by Kohsuke Kawaguchi .
            jglick Jesse Glick made changes -
            Field Original Value New Value
            Link This issue blocks JENKINS-43713 [ JENKINS-43713 ]
            jglick Jesse Glick made changes -
            Link This issue blocks JENKINS-41745 [ JENKINS-41745 ]
            jglick Jesse Glick made changes -
            Labels cli jetty cli jetty regression
            jglick Jesse Glick made changes -
            Remote Link This issue links to "discussion in PR 2795 (Web Link)" [ 17635 ]
            Hide
            jglick Jesse Glick added a comment -

            Does not seem to be reproducible using java -jar jenkins.war; not sure what might be different in the bundled Jetty. Needs investigation.

            Show
            jglick Jesse Glick added a comment - Does not seem to be reproducible using java -jar jenkins.war ; not sure what might be different in the bundled Jetty. Needs investigation.
            jglick Jesse Glick made changes -
            Priority Critical [ 2 ] Major [ 3 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            Jenkinsfile
            test/src/test/java/hudson/cli/CLIActionTest.java
            http://jenkins-ci.org/commit/jenkins/810a21534d5183055f303fd6f2bf69eb85e19367
            Log:
            JENKINS-46659 Suppressing failing CLIActionTest.interleavedStdio.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: Jenkinsfile test/src/test/java/hudson/cli/CLIActionTest.java http://jenkins-ci.org/commit/jenkins/810a21534d5183055f303fd6f2bf69eb85e19367 Log: JENKINS-46659 Suppressing failing CLIActionTest.interleavedStdio.
            Hide
            olamy Olivier Lamy added a comment -

            Jesse Glick any stack trace?

            Show
            olamy Olivier Lamy added a comment - Jesse Glick any stack trace?
            Hide
            jglick Jesse Glick added a comment -

            Olivier Lamy

            Sep 08, 2017 5:30:04 PM hudson.cli.PlainCLIProtocol$EitherSide$Reader run
            FINE: ignoring idle timeout, perhaps from Jetty
            java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms
            	at org.eclipse.jetty.server.HttpInput$ErrorState.noContent(HttpInput.java:1047)
            	at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:307)
            	at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:250)
            	at org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:58)
            	at hudson.cli.FlightRecorderInputStream.read(FlightRecorderInputStream.java:88)
            	at java.io.DataInputStream.readInt(DataInputStream.java:387)
            	at hudson.cli.PlainCLIProtocol$EitherSide$Reader.run(PlainCLIProtocol.java:111)
            Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms
            	at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166)
            	at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50)
            	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            	at java.lang.Thread.run(Thread.java:748)
            

            The stack trace on its own is not particularly interesting because we got the same in Jetty 8. The difference is that 8 allowed Jenkins to catch the error but then go right back to reading from the channel, whereas 9 seems to repeatedly throw the same error—it is not recoverable.

            Probably PlainCLIProtocol needs to be extended to include a ping packet type.

            Show
            jglick Jesse Glick added a comment - Olivier Lamy Sep 08, 2017 5:30:04 PM hudson.cli.PlainCLIProtocol$EitherSide$Reader run FINE: ignoring idle timeout, perhaps from Jetty java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms at org.eclipse.jetty.server.HttpInput$ErrorState.noContent(HttpInput.java:1047) at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:307) at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:250) at org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:58) at hudson.cli.FlightRecorderInputStream.read(FlightRecorderInputStream.java:88) at java.io.DataInputStream.readInt(DataInputStream.java:387) at hudson.cli.PlainCLIProtocol$EitherSide$Reader.run(PlainCLIProtocol.java:111) Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166) at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) The stack trace on its own is not particularly interesting because we got the same in Jetty 8. The difference is that 8 allowed Jenkins to catch the error but then go right back to reading from the channel, whereas 9 seems to repeatedly throw the same error—it is not recoverable. Probably PlainCLIProtocol needs to be extended to include a ping packet type.
            jglick Jesse Glick made changes -
            Assignee Jesse Glick [ jglick ]
            jglick Jesse Glick made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            jglick Jesse Glick made changes -
            Labels cli jetty regression cli jetty lts-candidate regression
            jglick Jesse Glick made changes -
            Remote Link This issue links to "PR 3011 (Web Link)" [ 17653 ]
            jglick Jesse Glick made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            cli/src/main/java/hudson/cli/CLI.java
            cli/src/main/java/hudson/cli/PlainCLIProtocol.java
            test/src/test/java/hudson/cli/CLIActionTest.java
            http://jenkins-ci.org/commit/jenkins/dc06c3f03b026c8faafa3e080f3a307c97f4fd42
            Log:
            [FIXED JENKINS-46659] Avoid Jetty timeouts by sending a ping-like packet every 10s while a CLI command is running.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: cli/src/main/java/hudson/cli/CLI.java cli/src/main/java/hudson/cli/PlainCLIProtocol.java test/src/test/java/hudson/cli/CLIActionTest.java http://jenkins-ci.org/commit/jenkins/dc06c3f03b026c8faafa3e080f3a307c97f4fd42 Log: [FIXED JENKINS-46659] Avoid Jetty timeouts by sending a ping-like packet every 10s while a CLI command is running.
            Hide
            danielbeck Daniel Beck added a comment -

            Merged towards 2.80.

            Show
            danielbeck Daniel Beck added a comment - Merged towards 2.80.
            danielbeck Daniel Beck made changes -
            Status In Review [ 10005 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            olivergondza Oliver Gondža made changes -
            Labels cli jetty lts-candidate regression 2.73.3-fixed cli jetty regression
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            cli/src/main/java/hudson/cli/CLI.java
            cli/src/main/java/hudson/cli/PlainCLIProtocol.java
            http://jenkins-ci.org/commit/jenkins/dda5903c99a46df0d70b7d143d4fd14389bd433b
            Log:
            [FIXED JENKINS-46659] Avoid Jetty timeouts by sending a ping-like packet every 10s while a CLI command is running.

            (cherry picked from commit dc06c3f03b026c8faafa3e080f3a307c97f4fd42)

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: cli/src/main/java/hudson/cli/CLI.java cli/src/main/java/hudson/cli/PlainCLIProtocol.java http://jenkins-ci.org/commit/jenkins/dda5903c99a46df0d70b7d143d4fd14389bd433b Log: [FIXED JENKINS-46659] Avoid Jetty timeouts by sending a ping-like packet every 10s while a CLI command is running. (cherry picked from commit dc06c3f03b026c8faafa3e080f3a307c97f4fd42)
            jamesdumay James Dumay made changes -
            Remote Link This issue links to "CloudBees Internal PRE-17 (Web Link)" [ 18238 ]

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                jglick Jesse Glick
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: