Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-53844

Each new HTTP query starts an infinite loop, burning CPU

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: next-executions-plugin
    • Labels:
    • Environment:
      Jenkins: 2.141
      OS: Ubuntu 16.04.5 LTS
      OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-0ubuntu0.16.04.1-b13)
    • Similar Issues:

      Description

      Not (yet) sure what has triggered this, but from now on every HTTP request to jenkins start a new never ending thread which burns CPU (one query: 100%, another one: 200% etc).

      After acquring a thread dump, all threads have the same signature (the web URL doesn't seem to matter, it's the same even with getting /threadDump):

      "Handling GET / from 1.2.3.4 : qtp849460928-26 View/index.jelly View/sidepanel.jelly NextExecutionsWidget/index.jelly" #26 prio=5 os_prio=0 tid=0x00007f5258559800 nid=0x18b75 runnable [0x00007f5228804000]
      java.lang.Thread.State: RUNNABLE
      at java.util.GregorianCalendar.computeTime(GregorianCalendar.java:2657)
      at java.util.Calendar.updateTime(Calendar.java:3393)
      at java.util.Calendar.getTimeInMillis(Calendar.java:1782)
      at java.util.Calendar.getMillisOf(Calendar.java:3410)
      at java.util.Calendar.compareTo(Calendar.java:2784)
      at hudson.scheduler.CronTab.ceil(CronTab.java:338)
      at hudson.scheduler.CronTab.ceil(CronTab.java:322)
      at hudson.plugins.nextexecutions.utils.NextExecutionsUtils.getNextBuild(NextExecutionsUtils.java:56)
      at hudson.plugins.nextexecutions.NextExecutionsWidget.getBuilds(NextExecutionsWidget.java:80)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
      at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
      at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
      at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
      at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
      at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
      at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
      at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
      at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
      at org.apache.commons.jelly.expression.ExpressionSupport.evaluateRecurse(ExpressionSupport.java:61)
      at org.apache.commons.jelly.expression.ExpressionSupport.evaluateAsIterator(ExpressionSupport.java:94)
      at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:89)
      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
      at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
      at org.apache.commons.jelly.tags.core.OtherwiseTag.doTag(OtherwiseTag.java:41)
      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
      at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38)
      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
      at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
      at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:147)
      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
      at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)
      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
      at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
      at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:147)
      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
      at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
      at org.kohsuke.stapler.jelly.CompressTag.doTag(CompressTag.java:44)
      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
      at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
      at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
      at org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(JellyClassTearOff.java:112)
      at org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:140)
      at org.kohsuke.stapler.IndexViewDispatcher.dispatch(IndexViewDispatcher.java:30)
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:734)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:864)
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:801)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:864)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:668)
      at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
      at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
      at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
      at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:64)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
      at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
      at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
      at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
      at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
      at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
      at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
      at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
      at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
      at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
      at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
      at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
      at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
      at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
      at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      at org.eclipse.jetty.server.Server.handle(Server.java:531)
      at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
      at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
      at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
      at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
      at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
      at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
      at java.lang.Thread.run(Thread.java:748)

        Attachments

          Activity

          Hide
          oleg_nenashev Oleg Nenashev added a comment -
          at hudson.plugins.nextexecutions.utils.NextExecutionsUtils.getNextBuild(NextExecutionsUtils.java:56)
          at hudson.plugins.nextexecutions.NextExecutionsWidget.getBuilds(NextExecutionsWidget.java:80)
          

          Whatever is happening there, it's caused by the Next Executions plugin. It tries to process all items in the view without any caching: https://github.com/jenkinsci/next-executions-plugin/blob/master/src/main/java/hudson/plugins/nextexecutions/NextExecutionsWidget.java#L54-L109

           

          Show
          oleg_nenashev Oleg Nenashev added a comment - at hudson.plugins.nextexecutions.utils.NextExecutionsUtils.getNextBuild(NextExecutionsUtils.java:56) at hudson.plugins.nextexecutions.NextExecutionsWidget.getBuilds(NextExecutionsWidget.java:80) Whatever is happening there, it's caused by the Next Executions plugin. It tries to process all items in the view without any caching: https://github.com/jenkinsci/next-executions-plugin/blob/master/src/main/java/hudson/plugins/nextexecutions/NextExecutionsWidget.java#L54-L109  
          Hide
          achagani Aslam Chagani added a comment -

          We ran into this issue as well and disabling the plugin as Attila Nagy suggested fixed the issue for now. Looks like it's when it's trying to run GregorianCalendar.computeTime() it somehow enters the infinite loop

          Show
          achagani Aslam Chagani added a comment - We ran into this issue as well and disabling the plugin as Attila Nagy suggested fixed the issue for now. Looks like it's when it's trying to run GregorianCalendar.computeTime() it somehow enters the infinite loop
          Hide
          dshvedchenko Denis Shvedchenko added a comment - - edited

          Hello, it was fixed in jenkins core 2.151 . NextExecution uses core CronTab class method. https://jenkins.io/security/advisory/2018-12-05/.

          so after upgrading you Jenkins you will be able to use those plugin again 

          Show
          dshvedchenko Denis Shvedchenko added a comment - - edited Hello, it was fixed in jenkins core 2.151 . NextExecution uses core CronTab class method.  https://jenkins.io/security/advisory/2018-12-05/. so after upgrading you Jenkins you will be able to use those plugin again 

            People

            • Assignee:
              Unassigned
              Reporter:
              bra Attila Nagy
            • Votes:
              2 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated: