Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38720

500 Server Error when notifying master of new mercurial commit: "Response header too large"

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: mercurial-plugin
    • Labels:
    • Environment:
      Jenkins 2.24, Linux vmkvmx01 2.6.32-431.29.2.el6.x86_64 #1 SMP Sun Jul 27 15:55:46 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux
    • Similar Issues:

      Description

      Notifying Jenkins of a new mercurial commit via /mercurial/notifyCommit responds with a '500 Server Error'. It looks like the notification is accepted in the Jenkins logs but an exception is also raised. The logs also show all the job polling that is triggered, perhaps related to the issue?

      curl -v  http://jenkins.bld.cbr.internal.example.com:8080/mercurial/notifyCommit?url=ssh://hg@scm01.dev.cbr.internal.example.com/example-source
      *   Trying 192.168.12.1...
      * Connected to jenkins.bld.cbr.internal.example.com (192.168.12.1) port 8080 (#0)
      > GET /mercurial/notifyCommit?url=ssh://hg@scm01.dev.cbr.internal.example.com/example-source HTTP/1.1
      > Host: jenkins.bld.cbr.internal.example.com:8080
      > User-Agent: curl/7.43.0
      > Accept: */*
      > 
      < HTTP/1.1 500 Server Error
      < Connection: close
      < Server: Jetty(9.2.z-SNAPSHOT)
      < 
      * Closing connection 0
      

      Log snippet:

      Oct 05, 2016 4:26:47 PM hudson.plugins.mercurial.MercurialStatus handleNotifyCommit
      INFO: Scheduling example-source for refresh
      Oct 05, 2016 4:26:47 PM org.eclipse.jetty.util.log.JavaUtilLog warn
      WARNING: Commit failed
      java.io.IOException: Response header too large
      	at org.eclipse.jetty.http.HttpGenerator.generateResponse(HttpGenerator.java:402)
      	at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:655)
      	at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:246)
      	at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:208)
      	at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:471)
      	at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:763)
      	at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:801)
      	at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:147)
      	at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:470)
      	at org.eclipse.jetty.server.Iso88591HttpWriter.write(Iso88591HttpWriter.java:47)
      	at org.eclipse.jetty.server.HttpWriter.write(HttpWriter.java:71)
      	at java.io.Writer.write(Writer.java:157)
      	at java.io.PrintWriter.newLine(PrintWriter.java:480)
      	at java.io.PrintWriter.println(PrintWriter.java:629)
      	at java.io.PrintWriter.println(PrintWriter.java:740)
      	at hudson.plugins.mercurial.MercurialStatus$1.generateResponse(MercurialStatus.java:175)
      	at org.kohsuke.stapler.HttpResponseRenderer$Default.handleHttpResponse(HttpResponseRenderer.java:124)
      	at org.kohsuke.stapler.HttpResponseRenderer$Default.generateResponse(HttpResponseRenderer.java:69)
      	at org.kohsuke.stapler.Function.renderResponse(Function.java:119)
      	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:102)
      	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:124)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
      	at org.kohsuke.stapler.MetaClass$11.dispatch(MetaClass.java:380)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
      	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
      	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
      	at org.eclipse.jetty.server.Server.handle(Server.java:499)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
      	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
      	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.nio.BufferOverflowException
      	at java.nio.Buffer.nextPutIndex(Buffer.java:521)
      	at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:169)
      	at org.eclipse.jetty.http.HttpGenerator.putSanitisedName(HttpGenerator.java:1052)
      	at org.eclipse.jetty.http.HttpGenerator.putTo(HttpGenerator.java:1086)
      	at org.eclipse.jetty.http.HttpGenerator.generateHeaders(HttpGenerator.java:705)
      	at org.eclipse.jetty.http.HttpGenerator.generateResponse(HttpGenerator.java:387)
      	... 64 more
      
      
      

        Attachments

          Issue Links

            Activity

            Hide
            jglick Jesse Glick added a comment -

            John Default then file an issue in git-plugin linked to this one.

            Show
            jglick Jesse Glick added a comment - John Default then file an issue in git-plugin linked to this one.
            Hide
            default John Default added a comment -

            To remove the strangeness, I just ran into the same problem (500 Server Error,Commit failed,Response header too large) using only git repos. (Jenkins 2.46.3, Git plugin 3.3.0, Git-client plugin 2.4.6)

             

            Show
            default John Default added a comment - To remove the strangeness, I just ran into the same problem (500 Server Error,Commit failed,Response header too large) using only git repos. (Jenkins 2.46.3, Git plugin 3.3.0, Git-client plugin 2.4.6)  
            Hide
            jglick Jesse Glick added a comment -

            Similar code exists in git-plugin but strangely I can find no analogous issue.

            Show
            jglick Jesse Glick added a comment - Similar code exists in git-plugin but strangely I can find no analogous issue.
            Hide
            jglick Jesse Glick added a comment -

            Should it just be a cap with no warning?

            Maybe some kind of notation indicating that the list was truncated, so scripts relying on the header would fail predictably.

            Show
            jglick Jesse Glick added a comment - Should it just be a cap with no warning? Maybe some kind of notation indicating that the list was truncated, so scripts relying on the header would fail predictably.
            Hide
            sonneveldsmartward Nick Sonneveld added a comment -

            Jesse Glick Hrm, I wonder if you can generate the "Response header too large" exception while adding properties to the header. It looks like the size isn't checked until the header is flushed just before the body starts being written.

            I'm certainly not checking the results in the header but others might. Should it just be a cap with no warning? Or should there be a further header with number of projects triggered? (with a possible "truncated" property as well?)

            Show
            sonneveldsmartward Nick Sonneveld added a comment - Jesse Glick Hrm, I wonder if you can generate the "Response header too large" exception while adding properties to the header. It looks like the size isn't checked until the header is flushed just before the body starts being written. I'm certainly not checking the results in the header but others might. Should it just be a cap with no warning? Or should there be a further header with number of projects triggered? (with a possible "truncated" property as well?)
            Hide
            jglick Jesse Glick added a comment -

            Well, then you just have a huge number of jobs all using the same repository, I guess? So MercurialStatus needs to be patched to cap the number of job names it will enumerate in its header.

            Show
            jglick Jesse Glick added a comment - Well, then you just have a huge number of jobs all using the same repository, I guess? So MercurialStatus needs to be patched to cap the number of job names it will enumerate in its header.
            Hide
            sonneveldsmartward Nick Sonneveld added a comment -

            Jesse Glick hah, I know. It was my attempt to de-identify the logs. They are all unique job names however.

            Show
            sonneveldsmartward Nick Sonneveld added a comment - Jesse Glick hah, I know. It was my attempt to de-identify the logs. They are all unique job names however.
            Hide
            jglick Jesse Glick added a comment -
            INFO: Triggering polling of <example job name>
            

            looks pretty fishy.

            Show
            jglick Jesse Glick added a comment - INFO: Triggering polling of <example job name> looks pretty fishy.

              People

              • Assignee:
                Unassigned
                Reporter:
                sonneveldsmartward Nick Sonneveld
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: