Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-20640

Jenkins become very slow and had 'proxy error'

    Details

    • Similar Issues:

      Description

      We have to restart out hanging jenkins instance because of this issue. I'm not changing it to Critical priority.

      Jenkins version : 1.539

      Proxy Error
      
      The proxy server received an invalid response from an upstream server.
      The proxy server could not handle the request GET /monitoring.
      
      Reason: Error reading from remote server
      
      
      WARNING: /monitoring
      org.apache.commons.jelly.JellyTagException: jar:file:/tmp/jetty-127.0.0.1-8080-jenkins.war--any-/webapp/WEB-INF/lib/jenkins-core-1.539.jar!/lib/layout/layout.jelly:75:72: <st:include> org.eclipse.jetty.io.EofException
              at org.apache.commons.jelly.impl.TagScript.handleException(TagScript.java:726)
              at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:279)
              at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
              at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)
              at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
              at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
              at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
              at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
              at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
              at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
              at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
              at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
              at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81)
              at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
              at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
              at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:95)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:631)
              at hudson.init.impl.InstallUncaughtExceptionHandler$1.reportException(InstallUncaughtExceptionHandler.java:25)
              at org.kohsuke.stapler.compression.CompressionFilter.reportException(CompressionFilter.java:63)
              at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
              at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
              at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
              at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
              at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
              at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
              at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
              at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
              at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
              at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
              at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
              at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
              at org.eclipse.jetty.server.Server.handle(Server.java:370)
              at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
              at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
              at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
              at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
              at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
              at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
              at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
              at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
              at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at java.lang.Thread.run(Thread.java:679)
      Caused by: org.eclipse.jetty.io.EofException
              at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:914)
              at org.eclipse.jetty.http.AbstractGenerator.flush(AbstractGenerator.java:443)
              at org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.java:100)
              at org.eclipse.jetty.server.AbstractHttpConnection$Output.flush(AbstractHttpConnection.java:1101)
              at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:161)
              at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:107)
              at com.jcraft.jzlib.DeflaterOutputStream.deflate(DeflaterOutputStream.java:144)
              at com.jcraft.jzlib.DeflaterOutputStream.write(DeflaterOutputStream.java:102)
              at org.kohsuke.stapler.compression.FilterServletOutputStream.write(FilterServletOutputStream.java:31)
              at org.kohsuke.stapler.jelly.DefaultScriptInvoker$1.write(DefaultScriptInvoker.java:99)
              at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:220)
              at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:290)
              at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:294)
              at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:140)
              at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
              at java.io.BufferedWriter.flush(BufferedWriter.java:253)
              at org.dom4j.io.XMLWriter.flush(XMLWriter.java:276)
              at org.apache.commons.jelly.XMLOutput.flush(XMLOutput.java:127)
              at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:271)
              ... 46 more
      Caused by: java.nio.channels.ClosedChannelException
              at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:146)
              at sun.nio.ch.SocketChannelImpl.write0(SocketChannelImpl.java:373)
              at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:406)
              at java.nio.channels.SocketChannel.write(SocketChannel.java:384)
              at org.eclipse.jetty.io.nio.ChannelEndPoint.gatheringFlush(ChannelEndPoint.java:376)
              at org.eclipse.jetty.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:335)
              at org.eclipse.jetty.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:376)
              at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:838)
      

        Attachments

          Issue Links

            Activity

            Hide
            evernat evernat added a comment -

            This issue is perhaps related to the HTTP session issue.
            To be sure that you don't have the HTTP session issue, I suggest to upgrade to the monitoring plugin v1.48 if you use Jenkins v1.535 or later.

            Show
            evernat evernat added a comment - This issue is perhaps related to the HTTP session issue. To be sure that you don't have the HTTP session issue, I suggest to upgrade to the monitoring plugin v1.48 if you use Jenkins v1.535 or later.
            Hide
            samxiao Sam Xiao added a comment -

            I don't think these are monitor plugin specific bugs, I've seen the proxy errors almost every endpoints. I think there's an upstream bug somewhere caused this, and only when Jenkins is under heavy load.

            Show
            samxiao Sam Xiao added a comment - I don't think these are monitor plugin specific bugs, I've seen the proxy errors almost every endpoints. I think there's an upstream bug somewhere caused this, and only when Jenkins is under heavy load.
            Hide
            docwhat Christian Höltje added a comment -

            @evernat Do you have a link to the bug about HTTP sessions? I'm seeing problems (with all endpoints) with both our 1.532.1 Jenkins servers.

            Show
            docwhat Christian Höltje added a comment - @evernat Do you have a link to the bug about HTTP sessions? I'm seeing problems (with all endpoints) with both our 1.532.1 Jenkins servers.
            Hide
            evernat evernat added a comment -

            @Christian Höltje
            The bug with HTTP sessions is JENKINS-20532
            It happened since Jenkins v1.535 with the change to Jetty.
            The bug is fixed since the Monitoring plugin v1.48

            Show
            evernat evernat added a comment - @Christian Höltje The bug with HTTP sessions is JENKINS-20532 It happened since Jenkins v1.535 with the change to Jetty. The bug is fixed since the Monitoring plugin v1.48
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Duplicates JENKINS-20074 (Fixed)

            Show
            oleg_nenashev Oleg Nenashev added a comment - Duplicates JENKINS-20074 (Fixed)

              People

              • Assignee:
                Unassigned
                Reporter:
                samxiao Sam Xiao
              • Votes:
                2 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: