Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-10524

winstone.ClientSocketException: Failed to write to client

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: core
    • Labels:
    • Environment:
      Windows Server 2003, JDK 1.6.0_23_64, Jenkins 1.415
    • Similar Issues:

      Description

      Slaves are connected via JNLP

      winstone.ClientSocketException: Failed to write to client
      at winstone.ClientOutputStream.write(ClientOutputStream.java:41)
      at winstone.WinstoneOutputStream.commit(WinstoneOutputStream.java:181)
      at winstone.WinstoneOutputStream.commit(WinstoneOutputStream.java:123)
      at winstone.WinstoneOutputStream.write(WinstoneOutputStream.java:112)
      at net.bull.javamelody.FilterServletOutputStream.write(FilterServletOutputStream.java:70)
      at net.bull.javamelody.CounterResponseStream.write(CounterResponseStream.java:83)
      at org.kohsuke.stapler.Stapler.serveStaticResource(Stapler.java:410)
      at org.kohsuke.stapler.Stapler.serveStaticResource(Stapler.java:241)
      at org.kohsuke.stapler.Stapler.service(Stapler.java:149)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:45)
      at winstone.ServletConfiguration.execute(ServletConfiguration.java:249)
      at winstone.RequestDispatcher.forward(RequestDispatcher.java:335)
      at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:378)
      at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:94)
      at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:157)
      at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:131)
      at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:82)
      at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84)
      at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:97)
      at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:52)
      at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:97)
      at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:65)
      at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:97)
      at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:86)
      at winstone.FilterConfiguration.execute(FilterConfiguration.java:195)
      at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368)
      at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47)
      at winstone.FilterConfiguration.execute(FilterConfiguration.java:195)
      at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
      at winstone.FilterConfiguration.execute(FilterConfiguration.java:195)
      at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368)
      at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
      at winstone.FilterConfiguration.execute(FilterConfiguration.java:195)
      at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368)
      at winstone.RequestDispatcher.forward(RequestDispatcher.java:333)
      at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:244)
      at winstone.RequestHandlerThread.run(RequestHandlerThread.java:150)
      at java.lang.Thread.run(Thread.java:662)
      Caused by: java.net.SocketException: Software caused connection abort: socket write error
      at java.net.SocketOutputStream.socketWrite0(Native Method)
      at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
      at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
      at winstone.ClientOutputStream.write(ClientOutputStream.java:39)
      ... 55 more

        Attachments

          Issue Links

            Activity

            robsimon robsimon created issue -
            Hide
            kohsuke Kohsuke Kawaguchi added a comment -

            The exception indicates that the HTTP client has shut down the connection without fully reading the response, which is actually quite normal.

            In Jenkins, the code is written such that we simply swallow this exception, but if you are seeing it in console, it must be that someone is catching it and dumping it. I suspect the monitoring plugin is causing this.

            Show
            kohsuke Kohsuke Kawaguchi added a comment - The exception indicates that the HTTP client has shut down the connection without fully reading the response, which is actually quite normal. In Jenkins, the code is written such that we simply swallow this exception, but if you are seeing it in console, it must be that someone is catching it and dumping it. I suspect the monitoring plugin is causing this.
            Hide
            jglick Jesse Glick added a comment -

            I have been tormented by similar exceptions (usually java.net.SocketException: Connection reset, but sometimes also java.io.IOException: Broken pipe from Jetty in 1.457 IIRC), which pollute the log file with junk.

            org.kohsuke.stapler.compression.CompressionFilter.reportException is what actually sends the log record. Not sure what “code is written such that we simply swallow this exception”.

            Unfortunately I have not yet managed to reproduce the problem in a controlled environment. Have tried installing a plugin with

            @Extension
            public class XXX extends PageDecorator {
                public void sleep() throws InterruptedException, IOException {
                    Stapler.getCurrentResponse().flushBuffer();
                    Thread.sleep(5000);
                }
            }
            

            and a matching footer.jelly

            <?xml version="1.0" encoding="UTF-8"?>
            <?jelly escape-by-default='true'?>
            <j:jelly xmlns:j="jelly:core" xmlns:st="jelly:stapler" xmlns:d="jelly:define" xmlns:i="jelly:fmt">
                <hr/>
                Sleeping…
                ${it.sleep()}
                done.
            </j:jelly>
            

            which makes a page take a long time to load, but halting it halfway prints nothing to the Jenkins console. Installing the Monitoring plugin changes nothing. Tried Jenkins trunk, 1.480.3, 1.466.2. Tried Stop button in Chrome, as well as running curl and interrupting the process before the page finishes loading.

            So if I can ever figure out how to reproduce this, I will try to patch reportException to check for exceptions (incl. nested causes) that seem to indicate a broken socket, and if so not send them to LOGGER (or use FINE rather than WARNING).

            Show
            jglick Jesse Glick added a comment - I have been tormented by similar exceptions (usually java.net.SocketException: Connection reset , but sometimes also java.io.IOException: Broken pipe from Jetty in 1.457 IIRC), which pollute the log file with junk. org.kohsuke.stapler.compression.CompressionFilter.reportException is what actually sends the log record. Not sure what “code is written such that we simply swallow this exception”. Unfortunately I have not yet managed to reproduce the problem in a controlled environment. Have tried installing a plugin with @Extension public class XXX extends PageDecorator { public void sleep() throws InterruptedException, IOException { Stapler.getCurrentResponse().flushBuffer(); Thread .sleep(5000); } } and a matching footer.jelly <?xml version= "1.0" encoding= "UTF-8" ?> <?jelly escape-by-default= 'true' ?> <j:jelly xmlns:j = "jelly:core" xmlns:st = "jelly:stapler" xmlns:d = "jelly:define" xmlns:i = "jelly:fmt" > <hr/> Sleeping… ${it.sleep()} done. </j:jelly> which makes a page take a long time to load, but halting it halfway prints nothing to the Jenkins console. Installing the Monitoring plugin changes nothing. Tried Jenkins trunk, 1.480.3, 1.466.2. Tried Stop button in Chrome, as well as running curl and interrupting the process before the page finishes loading. So if I can ever figure out how to reproduce this, I will try to patch reportException to check for exceptions (incl. nested causes) that seem to indicate a broken socket, and if so not send them to LOGGER (or use FINE rather than WARNING ).
            Hide
            jglick Jesse Glick added a comment -

            Also java.net.SocketException: Broken pipe from Tomcat.

            Show
            jglick Jesse Glick added a comment - Also java.net.SocketException: Broken pipe from Tomcat.
            jglick Jesse Glick made changes -
            Field Original Value New Value
            Assignee Jesse Glick [ jglick ]
            hardwickj James Hardwick made changes -
            Link This issue is related to JENKINS-14825 [ JENKINS-14825 ]
            Hide
            hardwickj James Hardwick added a comment -

            I've been trying to find a pattern as well for reproducing this (and JENKINS-14825) but have had no luck. The randomness is making it a really stickler.

            I'll keep trying and hopefully we can come up with something. Thanks for looking into it Jesse!

            Show
            hardwickj James Hardwick added a comment - I've been trying to find a pattern as well for reproducing this (and JENKINS-14825 ) but have had no luck. The randomness is making it a really stickler. I'll keep trying and hopefully we can come up with something. Thanks for looking into it Jesse!
            Hide
            hardwickj James Hardwick added a comment -

            @Kohsuke I also suspected the monitoring plugin, so I uninstalled it entirely and removed JENKINS_HOME/monitoring. Still no luck.

            The odd thing about this is that it seems to result in HTTP 404 errors on the front end, sometimes for an entire page and other times just for certain ajax calls. I'd expect a 500ish error.

            Show
            hardwickj James Hardwick added a comment - @Kohsuke I also suspected the monitoring plugin, so I uninstalled it entirely and removed JENKINS_HOME/monitoring. Still no luck. The odd thing about this is that it seems to result in HTTP 404 errors on the front end, sometimes for an entire page and other times just for certain ajax calls. I'd expect a 500ish error.
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-14825 [ JENKINS-14825 ]
            jglick Jesse Glick made changes -
            Link This issue is related to JENKINS-14825 [ JENKINS-14825 ]
            Hide
            jglick Jesse Glick added a comment -

            @hardwickj: this and the duplicate issue seem to be mixing up several things.

            One is the issue reported in the summary, which is that a broken client connection results in a noisy stack trace in the Jenkins log file, which it should not, since this is a normal condition for an HTTP-based server application.

            You report some 404s, which are probably some unrelated bug whose cause I cannot speculate on.

            And others in JENKINS-14825 report extremely long delays in page loads under some circumstances. Such delays probably trigger this bug, but are caused by something unrelated—which may well be different in each case. Only a thread dump can diagnose why a page load is hanging.

            Show
            jglick Jesse Glick added a comment - @hardwickj: this and the duplicate issue seem to be mixing up several things. One is the issue reported in the summary, which is that a broken client connection results in a noisy stack trace in the Jenkins log file, which it should not, since this is a normal condition for an HTTP-based server application. You report some 404s, which are probably some unrelated bug whose cause I cannot speculate on. And others in JENKINS-14825 report extremely long delays in page loads under some circumstances. Such delays probably trigger this bug, but are caused by something unrelated—which may well be different in each case. Only a thread dump can diagnose why a page load is hanging.
            Hide
            comxudong2012 xu dong added a comment - - edited

            Same here on Windows xp X86 SP2 with java version "1.7.0_11", Jenkins ver: 1.488 ,build-pipeline: 1.3.4-snapshot,run by java -jar jenkins.war

            Show
            comxudong2012 xu dong added a comment - - edited Same here on Windows xp X86 SP2 with java version "1.7.0_11", Jenkins ver: 1.488 ,build-pipeline: 1.3.4-snapshot,run by java -jar jenkins.war
            comxudong2012 xu dong made changes -
            Labels build-pipeline
            Hide
            hardwickj James Hardwick added a comment -

            @Jesse: we discovered this was being caused when we accessed our Jenkins instance through a proxy. After some digging we stumbled across https://wiki.jenkins-ci.org/display/JENKINS/Running+Jenkins+behind+Apache which mentions setting the proxy-nokeepalive value in our apache config. After setting that value, all our issues went away. The 404's and stack trace in the jenkins log were related, albeit loosely. When the proxied connection broke down, apache would return a 404, and Jenkins would spit out the ClientSocketException.

            We discovered this when one of our developers mentioned he always accessed Jenkins directly rather than through our proxy and he had never had issues. With a little wget magic we saw he was absolutely right, which set us on the right path.

            Here's hoping this helps everyone else out...

            Show
            hardwickj James Hardwick added a comment - @Jesse: we discovered this was being caused when we accessed our Jenkins instance through a proxy. After some digging we stumbled across https://wiki.jenkins-ci.org/display/JENKINS/Running+Jenkins+behind+Apache which mentions setting the proxy-nokeepalive value in our apache config. After setting that value, all our issues went away. The 404's and stack trace in the jenkins log were related, albeit loosely. When the proxied connection broke down, apache would return a 404, and Jenkins would spit out the ClientSocketException. We discovered this when one of our developers mentioned he always accessed Jenkins directly rather than through our proxy and he had never had issues. With a little wget magic we saw he was absolutely right, which set us on the right path. Here's hoping this helps everyone else out...
            Hide
            jglick Jesse Glick added a comment -

            @hardwickj: very interesting, thanks for noting this. Might help reproduce the stack trace, and is a good tip for people encountering odd errors. (I am wondering whether some incidences of JENKINS-16474 are related.)

            Bizarre that Apache would produce the totally inappropriate 404 status code for this situation though; I would have expected a 504 here! If you can reproduce that, consider filing a bug for Apache.

            Show
            jglick Jesse Glick added a comment - @hardwickj: very interesting, thanks for noting this. Might help reproduce the stack trace, and is a good tip for people encountering odd errors. (I am wondering whether some incidences of JENKINS-16474 are related.) Bizarre that Apache would produce the totally inappropriate 404 status code for this situation though; I would have expected a 504 here! If you can reproduce that, consider filing a bug for Apache.
            Hide
            dregin Bernard McKeever added a comment - - edited

            I've worked around this by adding Keepalive=On to the ProxyPass line in my VHost configuration.
            The above hasn't worked. Still seeing this issue.

            It seems to happen more often in chrome where I've disabled the browser's cache than firefox where I haven't...

            There seem to be a LOT of javascript files loaded on each page refresh, so this may have something to do with it?

            Show
            dregin Bernard McKeever added a comment - - edited I've worked around this by adding Keepalive=On to the ProxyPass line in my VHost configuration. The above hasn't worked. Still seeing this issue. It seems to happen more often in chrome where I've disabled the browser's cache than firefox where I haven't... There seem to be a LOT of javascript files loaded on each page refresh, so this may have something to do with it?
            Hide
            jglick Jesse Glick added a comment -

            Still no luck reproducing this with an Apache reverse proxy. Maybe browser-specific?

            Show
            jglick Jesse Glick added a comment - Still no luck reproducing this with an Apache reverse proxy. Maybe browser-specific?
            jglick Jesse Glick made changes -
            Labels build-pipeline
            Hide
            teilo James Nord added a comment -

            have the issue and we do not use a reverse proxy.

            Show
            teilo James Nord added a comment - have the issue and we do not use a reverse proxy.
            Hide
            hardwickj James Hardwick added a comment - - edited

            If it were a matter of too many javascript files being loaded than a lot more users would be having this problem.

            If you guys are running Jenkins behind an apache proxy, try just hitting jenkins directly for a day and see what your results are. This might require typing a longer URL and appending :8080 at the end, but definitely worth it for debugging reasons.

            All our problems went away once we figured out our apache proxying issue. We initially discovered the discrepancy by running the following curl command against jenkins, first through the apache proxy, and second against jenkins directly:

             
            #!/bin/bash
            
            for i in {1..100}
            do
              # via proxy
              wget --quiet --spider http://build/api/json?pretty=true
              if [ $? -ne 0 ]; then
                echo "wget failed via proxy"
              fi
            done
            
            for i in {1..100}
            do
              # bypassing the proxy
              wget --quiet --spider http://jenkins:8080/api/json?pretty=true
              if [ $? -ne 0 ]; then
                echo "wget failed directly"
              fi
            done
            

            If you run that and you only see "wget failed via proxy" but nothing directly, then it's your proxy configuration.

            Show
            hardwickj James Hardwick added a comment - - edited If it were a matter of too many javascript files being loaded than a lot more users would be having this problem. If you guys are running Jenkins behind an apache proxy, try just hitting jenkins directly for a day and see what your results are. This might require typing a longer URL and appending :8080 at the end, but definitely worth it for debugging reasons. All our problems went away once we figured out our apache proxying issue. We initially discovered the discrepancy by running the following curl command against jenkins, first through the apache proxy, and second against jenkins directly: #!/bin/bash for i in {1..100} do # via proxy wget --quiet --spider http: //build/api/json?pretty= true if [ $? -ne 0 ]; then echo "wget failed via proxy" fi done for i in {1..100} do # bypassing the proxy wget --quiet --spider http: //jenkins:8080/api/json?pretty= true if [ $? -ne 0 ]; then echo "wget failed directly" fi done If you run that and you only see "wget failed via proxy" but nothing directly, then it's your proxy configuration.
            Hide
            hardwickj James Hardwick added a comment -

            My point being, if you use a proxy at all and are having issues, give that a try first.

            Show
            hardwickj James Hardwick added a comment - My point being, if you use a proxy at all and are having issues, give that a try first.
            Hide
            hx_unbanned Linards L added a comment -

            Same here on 2k8r2 x64 sp1 @ jdk 1.7 @ v1.514.

            Show
            hx_unbanned Linards L added a comment - Same here on 2k8r2 x64 sp1 @ jdk 1.7 @ v1.514.
            Hide
            slaag Sebastian Laag added a comment -

            Same problem here. Using v1.516 with Windows Server 2008 R2, JDK 1.7 Update 21 64Bit.

            We use a master and 10 slaves. All connected without any proxy.

            If i request any of the master jenkins urls via VPN this exception occurs more often.

            Show
            slaag Sebastian Laag added a comment - Same problem here. Using v1.516 with Windows Server 2008 R2, JDK 1.7 Update 21 64Bit. We use a master and 10 slaves. All connected without any proxy. If i request any of the master jenkins urls via VPN this exception occurs more often.
            Hide
            ashex Ahmed Osman added a comment - - edited

            Experiencing this issue, while we are using a proxy we did not run into this until we configured matrix based security with the bitbucket oAuth plugin. With matrix based security in use, the error presented itself after logging out then back in. Once we switched aware from matrix based security the error went away.

            Show
            ashex Ahmed Osman added a comment - - edited Experiencing this issue, while we are using a proxy we did not run into this until we configured matrix based security with the bitbucket oAuth plugin. With matrix based security in use, the error presented itself after logging out then back in. Once we switched aware from matrix based security the error went away.
            Hide
            jglick Jesse Glick added a comment - - edited

            Jenkins 1.521 included https://github.com/stapler/stapler/commit/ac0b5683c4a3a45c3b987baf264674934470b668 which installs a new pretty exception handler that may affect the symptoms of this bug somehow.

            Show
            jglick Jesse Glick added a comment - - edited Jenkins 1.521 included https://github.com/stapler/stapler/commit/ac0b5683c4a3a45c3b987baf264674934470b668 which installs a new pretty exception handler that may affect the symptoms of this bug somehow.
            Hide
            jglick Jesse Glick added a comment -
            Show
            jglick Jesse Glick added a comment - https://github.com/stapler/stapler/commit/a6307f473e44aa4fd6a26275304d738dba5201c9 (once integrated into Jenkins core) might work.
            jglick Jesse Glick made changes -
            Labels lts-candidate
            jglick Jesse Glick made changes -
            Link This issue is related to JENKINS-18992 [ JENKINS-18992 ]
            Hide
            jglick Jesse Glick added a comment -

            7e8bbea3336dc65a10c2fd69a20beac20bc6fca3 in 1.528 should have integrated the purported fix.

            Show
            jglick Jesse Glick added a comment - 7e8bbea3336dc65a10c2fd69a20beac20bc6fca3 in 1.528 should have integrated the purported fix.
            jglick Jesse Glick made changes -
            Status Open [ 1 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            core/pom.xml
            http://jenkins-ci.org/commit/jenkins/fdbcfda5acd430a8f8bfb90d7def4351f7bfce57
            Log:
            [FIXED JENKINS-10524][FIXED JENKINS-19168] Stapler 1.218.

            (cherry picked from commit d814a6ff7f2e99c9ada722d17c33ca87b0bbea65)

            Conflicts:
            changelog.html
            core/pom.xml

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/pom.xml http://jenkins-ci.org/commit/jenkins/fdbcfda5acd430a8f8bfb90d7def4351f7bfce57 Log: [FIXED JENKINS-10524] [FIXED JENKINS-19168] Stapler 1.218. (cherry picked from commit d814a6ff7f2e99c9ada722d17c33ca87b0bbea65) Conflicts: changelog.html core/pom.xml
            olivergondza Oliver Gondža made changes -
            Labels lts-candidate 1.509.4-fixed
            jglick Jesse Glick made changes -
            Link This issue is related to JENKINS-20074 [ JENKINS-20074 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 140717 ] JNJira + In-Review [ 189216 ]

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                robsimon robsimon
              • Votes:
                10 Vote for this issue
                Watchers:
                22 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: