Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45661

NegativeArraySizeException at readFrom(ConsoleNote.java:240) while printing job's console output

    Details

    • Similar Issues:
    • Released As:
      2.170

      Description

      Hi everyone,

      this error shows up after some time when we open one of our jobs. Also jenkins display impossible big size of console output (about 200MB). We using jenkins 2.63, update to latest (2.70) is in progress and we'll see if can helps soon.

      A problem occurred while processing the request.
       Please check our bug tracker to see if a similar problem has already been reported.
       If it is already reported, please vote and put a comment on it to let us gauge the impact of the problem.
       If you think this is a new issue, please file a new issue.
       When you file an issue, make sure to add the entire stack trace, along with the version of Jenkins and relevant plugins.
       The users list might be also useful in understanding what has happened.
      Stack trace
      java.lang.NegativeArraySizeException
       at hudson.console.ConsoleNote.readFrom(ConsoleNote.java:240)
       at hudson.console.ConsoleAnnotationOutputStream.eol(ConsoleAnnotationOutputStream.java:109)
       at hudson.console.LineTransformationOutputStream.eol(LineTransformationOutputStream.java:60)
       at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:56)
       at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:74)
       at org.apache.commons.io.output.ProxyOutputStream.write(ProxyOutputStream.java:90)
       at org.kohsuke.stapler.framework.io.LargeText$HeadMark.moveTo(LargeText.java:314)
       at org.kohsuke.stapler.framework.io.LargeText.writeLogTo(LargeText.java:229)
       at hudson.console.AnnotatedLargeText.writeHtmlTo(AnnotatedLargeText.java:168)
       at hudson.console.AnnotatedLargeText.writeLogTo(AnnotatedLargeText.java:143)
       at org.kohsuke.stapler.framework.io.LargeText.doProgressText(LargeText.java:264)
       at hudson.console.AnnotatedLargeText.doProgressiveHtml(AnnotatedLargeText.java:91)
       at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
       at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:343)
       at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184)
       at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117)
       at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:129)
       at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
       at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
      Caused: javax.servlet.ServletException
       at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:765)
       at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
       at org.kohsuke.stapler.MetaClass$3.doDispatch(MetaClass.java:209)
       at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
       at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
       at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
       at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:374)
       at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
       at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
       at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:248)
       at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
       at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
       at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
       at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
       at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
       at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
       at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
       at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
       at com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47)
       at  hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
       at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:230)
       at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:202)
       at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:85)
       at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:112)
       at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
       at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:138)
       at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
       at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
       at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
       at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
       at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
       at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
       at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
       at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
       at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
       at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
       at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
       at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
       at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
       at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
       at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:92)
       at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
       at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
       at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
       at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
       at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
       at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
       at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
       at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
       at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
       at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
       at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
       at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
       at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
       at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
       at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
       at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
       at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
       at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
       at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
       at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
       at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
       at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
       at org.eclipse.jetty.server.Server.handle(Server.java:564)
       at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317)
       at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
       at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
       at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
       at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
       at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128)
       at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222)
       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294)
       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:199)
       at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       at java.lang.Thread.run(Thread.java:745)
      Page generated: Jul 19, 2017 1:24:33 PM UTCREST APIJenkins ver. 2.63

      Common steps to reproduce is to open console output of running jobs, then wait a few minutes while output is still appending new data. Later i can see this exception (screenshot-1), if i open plain text console output i can see this (we using salt).

      ----------
                ID: /etc/sudoers.d/90-salt-user-xxxxx
          Function: file.managed
            Result: True
           Comment: File /etc/sudoers.d/90-salt-user-xxxxx updated
           Started: 11:16:04.538259
          Duration: 127.515 ms
           Changes:   
                    ----------
                    diff:
                        New file
                    mode:
                        0440
      ----------
                ID: system_user_xxxxx
          Function: user.present
              Name: xxxxx
            Result: True
           Comment: New user xxxxx created
           Started: 11:16:04.666108
          Duration: 146.021 ms
           Changes:   
                    ----------
                    fullname:
                    gid:
                        1039
                    group�������������                    group�������������      

        Attachments

          Issue Links

            Activity

            Hide
            jakubjosef Jakub Josef added a comment -

            Version 2.71 is also affected.

            Show
            jakubjosef Jakub Josef added a comment - Version 2.71 is also affected.
            Hide
            jglick Jesse Glick added a comment -

            The root cause is a corrupted ConsoleNote serialized into the build log. One observed case was tracked down to NewNodeConsoleNote (from workflow-job) being printed and correctly using synchronization, but TimestampNote (from timestamper) neglecting to use synchronization and thus intermingling the escape sequences for the two notes. This particular case should be solved simply by upgrading the timestamper plugin to pick up JENKINS-48344, as new versions do not use notes for Pipeline builds. Other instances of the bug could be attributed to other root causes.

            Show
            jglick Jesse Glick added a comment - The root cause is a corrupted ConsoleNote serialized into the build log. One observed case was tracked down to NewNodeConsoleNote (from workflow-job ) being printed and correctly using synchronization, but TimestampNote (from timestamper ) neglecting to use synchronization and thus intermingling the escape sequences for the two notes. This particular case should be solved simply by upgrading the timestamper plugin to pick up JENKINS-48344 , as new versions do not use notes for Pipeline builds. Other instances of the bug could be attributed to other root causes.
            Hide
            jglick Jesse Glick added a comment -

            For LTS purposes it would suffice to cherry-pick just 0082b67a7f2c0a1abca07e6f746360568a57a768; the other two commits assist in diagnosis but are not required to solve the main user problem.

            Show
            jglick Jesse Glick added a comment - For LTS purposes it would suffice to cherry-pick just 0082b67a7f2c0a1abca07e6f746360568a57a768 ; the other two commits assist in diagnosis but are not required to solve the main user problem.
            Hide
            jglick Jesse Glick added a comment -

            Looks like Oliver Gondža missed my recommendation and backported the whole thing, as ed84746e22e7f979ed1b8c69bf4c622471f0e39a. Probably OK.

            Show
            jglick Jesse Glick added a comment - Looks like Oliver Gondža missed my recommendation and backported the whole thing, as ed84746e22e7f979ed1b8c69bf4c622471f0e39a . Probably OK.
            Hide
            olivergondza Oliver Gondža added a comment -

            Correct, I backported both the fix and diagnostics intentionally.

            Show
            olivergondza Oliver Gondža added a comment - Correct, I backported both the fix and diagnostics intentionally.

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                jakubjosef Jakub Josef
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: