Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-23971

NumberFormatException when tailing node log (ssh unix slave, ec2)

    Details

    • Similar Issues:

      Description

      When tailing the log of a newly launched ssh unix slave, Jenkins suddenly stopped showing the log and instead reported:

      javax.servlet.ServletException: java.lang.NumberFormatException: For input string: "5584start=208664"
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:778)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
      	at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:210)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
      	at org.kohsuke.stapler.MetaClass$12.dispatch(MetaClass.java:390)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
      	at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:210)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:631)
      	at org.kohsuke.stapler.Stapler.service(Stapler.java:225)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
      	at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
      	at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:117)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
      	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:79)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
      	at org.eclipse.jetty.server.Server.handle(Server.java:370)
      	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
      	at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
      	at org.eclipse.jetty.ajp.Ajp13Connection.access$2900(Ajp13Connection.java:45)
      	at org.eclipse.jetty.ajp.Ajp13Connection$RequestHandler.headerComplete(Ajp13Connection.java:237)
      	at org.eclipse.jetty.ajp.Ajp13Parser.parseNext(Ajp13Parser.java:505)
      	at org.eclipse.jetty.ajp.Ajp13Parser.parseAvailable(Ajp13Parser.java:158)
      	at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
      	at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
      	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:701)
      Caused by: java.lang.NumberFormatException: For input string: "5584start=208664"
      	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
      	at java.lang.Long.parseLong(Long.java:438)
      	at java.lang.Long.parseLong(Long.java:478)
      	at org.kohsuke.stapler.framework.io.LargeText.doProgressText(LargeText.java:256)
      	at hudson.console.AnnotatedLargeText.doProgressiveHtml(AnnotatedLargeText.java:91)
      	at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:622)
      	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
      	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
      	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
      	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:120)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
      	... 74 more
      

      as an "Oops!" report and stopped following the log.

      I've recently been seeing this happening a lot, and it seems new in Jenkins ver. 1.574-SNAPSHOT . I'm on git rev 17d9093.

        Attachments

          Activity

          Hide
          ringerc Craig Ringer added a comment -

          This doesn't seem to disrupt the ssh communication channel; slave/master communication continues and logs are captured as normal. Re-entering the node's page and opening the log again shows the rest of the log.

          Show
          ringerc Craig Ringer added a comment - This doesn't seem to disrupt the ssh communication channel; slave/master communication continues and logs are captured as normal. Re-entering the node's page and opening the log again shows the rest of the log.
          Hide
          danielbeck Daniel Beck added a comment -

          Are you behind a reverse proxy? If so, does this also happen when accessing Jenkins directly?

          Show
          danielbeck Daniel Beck added a comment - Are you behind a reverse proxy? If so, does this also happen when accessing Jenkins directly?
          Hide
          sdr984 Scott Roberts added a comment -

          We're running Jenkins version 1.564 and we just saw this for the first time today as well.

          Error while serving http://build.rctanalytics.com/job/Connection%20Broker/41/promotion/Production/promotionBuild/6/logText/progressiveHtml
          ...
          Caused by: java.lang.NumberFormatException: For input string: "1251start=24"
          at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
          at java.lang.Long.parseLong(Long.java:438)
          at java.lang.Long.parseLong(Long.java:478)
          at org.kohsuke.stapler.framework.io.LargeText.doProgressText(LargeText.java:256)
          at hudson.console.AnnotatedLargeText.doProgressiveHtml(AnnotatedLargeText.java:91)

          Our version is 1.564, but we have updated plugins if those might be the cause:

          Active Directory plugin
          This plugin enables authentication through Active Directory on Windows environment.
          1.38
          Downgrade to 1.37

          Ant Plugin
          This plugin adds Apache Ant support to Jenkins.
          1.2

          Artifactory Plugin
          This plugin allows deploying maven artifacts and build info to Artifactory.
          2.2.3
          Downgrade to 2.2.1

          Copy Artifact Plugin
          Adds a build step to copy artifacts from another project.
          1.31
          Downgrade to 1.30

          Credentials Plugin
          This plugin allows you to store credentials in Jenkins.
          1.15
          Downgrade to 1.15
          Unpin

          CVS Plug-in
          Integrates Jenkins with CVS version control system using a modified version of the Netbeans cvsclient.
          2.12
          Downgrade to 2.11
          Unpin

          Deploy to container Plugin
          This plugin allows you to deploy a war to a container after a successful build.
          Glassfish 3.x remote deployment
          1.10
          Downgrade to 1.9

          External Monitor Job Type Plugin
          Adds the ability to monitor the result of externally executed jobs.
          1.2
          Downgrade to 1.1
          Unpin

          GIT client plugin
          Shared library plugin for other Git related Jenkins plugins.
          1.10.0
          Downgrade to 1.9.2

          GIT plugin
          This plugin integrates GIT with Jenkins.
          2.2.2
          Downgrade to 2.2.1

          Git server plugin
          Allows Jenkins to act as a Git server.
          1.3
          Downgrade to 1.2

          Gradle plugin
          This plugin allows Jenkins to invoke Gradle build scripts directly.
          1.24
          Downgrade to 1.23

          Grails plugin
          This plugin supports building Grails applications from Jenkins.
          1.7

          Hudson Groovy builder
          This plugin executes Groovy code.
          1.19
          Downgrade to 1.18

          instant-messaging plugin
          This plugin provides abstract support for build notification via instant-messaging.
          1.29
          Downgrade to 1.28

          Jabber notifier plugin
          Sends build notifications to jabber contacts and/or chatrooms. Also allows control of builds via a jabber 'bot'.

          Note that the instant-messaging plugin 1.24 is a requirement for this plugin. Please make sure that it is installed, too!

          1.25

          Javadoc Plugin
          This plugin adds Javadoc support to Jenkins.
          1.1

          LDAP Plugin
          Security realm based on LDAP authentication.
          1.10.2
          Downgrade to 1.8
          Unpin

          Mailer Plugin
          This plugin allows you to configure email notifications. This is a break-out of the original core based email component.
          1.9
          Downgrade to 1.8
          Unpin

          MapDB API Plugin
          This plugin provides a shared dependency on the MapDB library so that other plugins can co-operate when using this library.
          1.0.1.0

          Matrix Authorization Strategy Plugin
          Offers matrix-based security authorization strategies (global and per-project).
          1.2
          Downgrade to 1.1
          Unpin

          Matrix Project Plugin
          Multi-configuration (matrix) project type.
          1.3
          Downgrade to 1.2
          Unpin

          Maven Integration plugin
          Jenkins plugin for building Maven 2/3 jobs via a special project type.
          2.5
          Downgrade to 2.3
          Unpin

          OWASP Markup Formatter Plugin
          Uses the OWASP Java HTML Sanitizer to allow safe-seeming HTML markup to be entered in project descriptions and the like.
          1.2
          Downgrade to 1.1
          Unpin

          PAM Authentication plugin
          Adds Unix Pluggable Authentication Module (PAM) support to Jenkins.
          1.1
          Downgrade to 1.0
          Unpin

          promoted builds plugin
          This plugin implements a "promoted build" feature where a build of one job can be marked as "promoted" when it passes certain criteria.
          2.17
          Downgrade to 2.13

          S3 publisher plugin
          This is a plugin to upload files to Amazon S3 buckets.
          0.6
          Downgrade to 0.5

          SCM API Plugin
          This plugin provides a new enhanced API for interacting with SCM systems.
          0.2

          Scriptler
          Scriptler allows you to store/edit/execute groovy scripts on any of the slaves/nodes... no need for copy paste groovy code anymore. Beside administer your scripts, Scritpler also provides a way to share scripts between users via hosted script catalogs on the internet.
          2.7
          Downgrade to 2.6.1

          SSH Agent Plugin
          This plugin allows you to provide SSH credentials to builds via a ssh-agent in Jenkins.
          1.4.1
          Downgrade to 1.3

          SSH Credentials Plugin
          This plugin allows you to store SSH credentials in Jenkins.
          1.7.1
          Downgrade to 1.6.1
          Unpin

          SSH Slaves plugin
          This plugin allows you to manage slaves running on *nix machines over SSH.
          1.6
          Downgrade to 1.2
          Unpin

          Subversion Plug-in
          This plugin adds the Subversion support (via SVNKit) to Jenkins.
          2.4.1
          Downgrade to 2.4
          Unpin

          Subversion Revert Plugin
          This plugin automatically performs Subversion reverting when the build status changes to unstable or failed.
          1.3

          Token Macro Plugin
          This plug-in adds reusable macro expansion capability for other plug-ins to use.
          1.10
          Downgrade to 1.8.1

          Translation Assistance plugin
          This plugin adds an additional dialog box in every page, which enables people to contribute localizations for the messages they are seeing in the current page.
          1.11
          Downgrade to 1.10
          Unpin

          Windows Slaves Plugin
          Allows you to connect to Windows machines and start slave agents on them.
          1.0

          Show
          sdr984 Scott Roberts added a comment - We're running Jenkins version 1.564 and we just saw this for the first time today as well. Error while serving http://build.rctanalytics.com/job/Connection%20Broker/41/promotion/Production/promotionBuild/6/logText/progressiveHtml ... Caused by: java.lang.NumberFormatException: For input string: "1251start=24" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Long.parseLong(Long.java:438) at java.lang.Long.parseLong(Long.java:478) at org.kohsuke.stapler.framework.io.LargeText.doProgressText(LargeText.java:256) at hudson.console.AnnotatedLargeText.doProgressiveHtml(AnnotatedLargeText.java:91) Our version is 1.564, but we have updated plugins if those might be the cause: Active Directory plugin This plugin enables authentication through Active Directory on Windows environment. 1.38 Downgrade to 1.37 Ant Plugin This plugin adds Apache Ant support to Jenkins. 1.2 Artifactory Plugin This plugin allows deploying maven artifacts and build info to Artifactory. 2.2.3 Downgrade to 2.2.1 Copy Artifact Plugin Adds a build step to copy artifacts from another project. 1.31 Downgrade to 1.30 Credentials Plugin This plugin allows you to store credentials in Jenkins. 1.15 Downgrade to 1.15 Unpin CVS Plug-in Integrates Jenkins with CVS version control system using a modified version of the Netbeans cvsclient. 2.12 Downgrade to 2.11 Unpin Deploy to container Plugin This plugin allows you to deploy a war to a container after a successful build. Glassfish 3.x remote deployment 1.10 Downgrade to 1.9 External Monitor Job Type Plugin Adds the ability to monitor the result of externally executed jobs. 1.2 Downgrade to 1.1 Unpin GIT client plugin Shared library plugin for other Git related Jenkins plugins. 1.10.0 Downgrade to 1.9.2 GIT plugin This plugin integrates GIT with Jenkins. 2.2.2 Downgrade to 2.2.1 Git server plugin Allows Jenkins to act as a Git server. 1.3 Downgrade to 1.2 Gradle plugin This plugin allows Jenkins to invoke Gradle build scripts directly. 1.24 Downgrade to 1.23 Grails plugin This plugin supports building Grails applications from Jenkins. 1.7 Hudson Groovy builder This plugin executes Groovy code. 1.19 Downgrade to 1.18 instant-messaging plugin This plugin provides abstract support for build notification via instant-messaging. 1.29 Downgrade to 1.28 Jabber notifier plugin Sends build notifications to jabber contacts and/or chatrooms. Also allows control of builds via a jabber 'bot'. Note that the instant-messaging plugin 1.24 is a requirement for this plugin. Please make sure that it is installed, too! 1.25 Javadoc Plugin This plugin adds Javadoc support to Jenkins. 1.1 LDAP Plugin Security realm based on LDAP authentication. 1.10.2 Downgrade to 1.8 Unpin Mailer Plugin This plugin allows you to configure email notifications. This is a break-out of the original core based email component. 1.9 Downgrade to 1.8 Unpin MapDB API Plugin This plugin provides a shared dependency on the MapDB library so that other plugins can co-operate when using this library. 1.0.1.0 Matrix Authorization Strategy Plugin Offers matrix-based security authorization strategies (global and per-project). 1.2 Downgrade to 1.1 Unpin Matrix Project Plugin Multi-configuration (matrix) project type. 1.3 Downgrade to 1.2 Unpin Maven Integration plugin Jenkins plugin for building Maven 2/3 jobs via a special project type. 2.5 Downgrade to 2.3 Unpin OWASP Markup Formatter Plugin Uses the OWASP Java HTML Sanitizer to allow safe-seeming HTML markup to be entered in project descriptions and the like. 1.2 Downgrade to 1.1 Unpin PAM Authentication plugin Adds Unix Pluggable Authentication Module (PAM) support to Jenkins. 1.1 Downgrade to 1.0 Unpin promoted builds plugin This plugin implements a "promoted build" feature where a build of one job can be marked as "promoted" when it passes certain criteria. 2.17 Downgrade to 2.13 S3 publisher plugin This is a plugin to upload files to Amazon S3 buckets. 0.6 Downgrade to 0.5 SCM API Plugin This plugin provides a new enhanced API for interacting with SCM systems. 0.2 Scriptler Scriptler allows you to store/edit/execute groovy scripts on any of the slaves/nodes... no need for copy paste groovy code anymore. Beside administer your scripts, Scritpler also provides a way to share scripts between users via hosted script catalogs on the internet. 2.7 Downgrade to 2.6.1 SSH Agent Plugin This plugin allows you to provide SSH credentials to builds via a ssh-agent in Jenkins. 1.4.1 Downgrade to 1.3 SSH Credentials Plugin This plugin allows you to store SSH credentials in Jenkins. 1.7.1 Downgrade to 1.6.1 Unpin SSH Slaves plugin This plugin allows you to manage slaves running on *nix machines over SSH. 1.6 Downgrade to 1.2 Unpin Subversion Plug-in This plugin adds the Subversion support (via SVNKit) to Jenkins. 2.4.1 Downgrade to 2.4 Unpin Subversion Revert Plugin This plugin automatically performs Subversion reverting when the build status changes to unstable or failed. 1.3 Token Macro Plugin This plug-in adds reusable macro expansion capability for other plug-ins to use. 1.10 Downgrade to 1.8.1 Translation Assistance plugin This plugin adds an additional dialog box in every page, which enables people to contribute localizations for the messages they are seeing in the current page. 1.11 Downgrade to 1.10 Unpin Windows Slaves Plugin Allows you to connect to Windows machines and start slave agents on them. 1.0
          Hide
          danielbeck Daniel Beck added a comment -

          If this happens fairly often (what web browser(s)?), try to keep the web developer console open, logging all requests. The following happens when viewing the log of a build in progress/slave log/promotion log:

          • The browser sends AJAX requests to Jenkins with the POST parameter 'start' to only load data that hasn't been received yet.
          • Jenkins response contains a HTTP header named 'X-Text-Size' which serves as the new starting point

          The cause for the issue is a garbled value sent by the client (it appears to be two values concatenated).

          It would be interesting to see where it goes wrong, so if it happens, check the log of past requests. Is it first the client to fail, or the server?

          Show
          danielbeck Daniel Beck added a comment - If this happens fairly often (what web browser(s)?), try to keep the web developer console open, logging all requests. The following happens when viewing the log of a build in progress/slave log/promotion log: The browser sends AJAX requests to Jenkins with the POST parameter 'start' to only load data that hasn't been received yet. Jenkins response contains a HTTP header named 'X-Text-Size' which serves as the new starting point The cause for the issue is a garbled value sent by the client (it appears to be two values concatenated). It would be interesting to see where it goes wrong, so if it happens, check the log of past requests. Is it first the client to fail, or the server?
          Hide
          ringerc Craig Ringer added a comment -

          Thanks for the tip Daniel. I'm using Chrome 35.0.1916.27 at the moment. I'll keep an eye on the dev console.

          Show
          ringerc Craig Ringer added a comment - Thanks for the tip Daniel. I'm using Chrome 35.0.1916.27 at the moment. I'll keep an eye on the dev console.
          Hide
          danielbeck Daniel Beck added a comment - - edited

          Any news?

          Would also be possible to look in the access logs for error responses to /progressivHtml to look for a commonality (same host, same browser, same job, ...).

          Show
          danielbeck Daniel Beck added a comment - - edited Any news? Would also be possible to look in the access logs for error responses to /progressivHtml to look for a commonality (same host, same browser, same job, ...).
          Hide
          danielbeck Daniel Beck added a comment -
          Show
          danielbeck Daniel Beck added a comment - Craig Ringer Ping!
          Hide
          danielbeck Daniel Beck added a comment -

          Resolving as Cannot Reproduce after a few months without update. Reopen if this happens again; make sure to provide the requested information.

          Show
          danielbeck Daniel Beck added a comment - Resolving as Cannot Reproduce after a few months without update. Reopen if this happens again; make sure to provide the requested information.

            People

            • Assignee:
              francisu Francis Upton
              Reporter:
              ringerc Craig Ringer
            • Votes:
              2 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: