Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-71627

static file downloads are truncated after a few minutes

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • winstone-jetty

      We have build jobs that download some large (100MB+) dependencies as build artifacts from our Jenkins CI. Sometimes while downloading, jenkins appears to close the connection a few minutes into the download, despite not having sent all the file data. At the time the connection is closed, the data transfer is still active and in progress (based on client side network monitoring).

      In the past this problem was resolved by adding `–httpKeepAliveTimeout=60000` to the `JENKINS_OPTS` however since updating Jenkins from 2.361.4 to 2.375.3 the problem has come back. It continues to occur in Jenkins 2.401.1

      We also changed our reverse proxy from Apache to HA Proxy and moved jenkins to a more powerful server, but the symptoms remain the same.

      No errors are recorded in the proxy log. The following error and stack trace occurs in the Jenkins log whenever the downloads are truncated (the file name will be different, but the error is `java.lang.IllegalStateException: ABORTED` .)

      2023-07-12 07:12:07.177+0000 [id=12206796]	WARNING	o.e.j.s.h.ContextHandler$Context#log: Error while serving https://resource.jenkins.[domain]/static-files/6CGfW9yV2UfO5Joz_H4QYs5MT85Qq7SmEom9NKp4IeoxNjg5MTQ1OTI2Mjc0OjEwOmJ1aWxkYWdlbnQ6am9iL2JhemVsX2RlcHMvam9iL21hc3Rlci9sYXN0U3VjY2Vzc2Z1bEJ1aWxkL2FydGlmYWN0/boost-1.80.0.tar.gz
	at java.base/java.lang.Thread.run(Thread.java:829)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$new$0(AdaptiveExecutionStrategy.java:140)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:272)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:385)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:416)
	at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505)
	at org.eclipse.jetty.server.Server.handle(Server.java:563)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1305)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1383)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:549)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
	at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:38)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:86)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:172)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:111)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:63)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:117)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221)
	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101)
	at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:145)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:54)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:94)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:160)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:160)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:64)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:129)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:81)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:157)
	at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:248)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:698)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770)
	at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:475)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770)
	at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:475)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770)
	at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:501)
	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:140)
	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:207)
	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:409)
	at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:401)
Caused: java.lang.reflect.InvocationTargetException
	at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:397)
	at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710)
	at jenkins.security.ResourceDomainRootAction$InternalResourceRequest.doDynamic(ResourceDomainRootAction.java:259)
	at javax.servlet.http.HttpServletResponseWrapper.sendError(HttpServletResponseWrapper.java:122)
	at jenkins.metrics.impl.MetricsFilter$StatusExposingServletResponse.sendError(MetricsFilter.java:153)
	at javax.servlet.http.HttpServletResponseWrapper.sendError(HttpServletResponseWrapper.java:122)
	at org.springframework.security.web.util.OnCommittedResponseWrapper.sendError(OnCommittedResponseWrapper.java:116)
	at javax.servlet.http.HttpServletResponseWrapper.sendError(HttpServletResponseWrapper.java:122)
	at javax.servlet.http.HttpServletResponseWrapper.sendError(HttpServletResponseWrapper.java:122)
	at org.eclipse.jetty.server.Response.sendError(Response.java:465)
	at org.eclipse.jetty.server.Response.sendError(Response.java:498)
	at org.eclipse.jetty.server.HttpChannelState.sendError(HttpChannelState.java:911)
java.lang.IllegalStateException: ABORTED

            Unassigned Unassigned
            sm__liam_baker Liam Baker
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: