Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55070

"Jenkins.instance is missing" error when restarting Jenkins

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Component/s: core
    • Labels:
    • Environment:
      Windows Server 2016
      Java 1.8.0_191-b12
      Jenkins 2.150.1
    • Similar Issues:

      Description

      Sometimes when I try to restart Jenkins (eg. after upgrading the plugins), it just starts returning HTTP 500 errors, and I need to restart the Jenkins service.

      When this happens, I see lots of these errors in the logs:

      Dec 07, 2018 12:03:45 AM org.eclipse.jetty.server.HttpChannel handleException
      WARNING: /
      java.lang.IllegalStateException: Jenkins.instance is missing. Read the documentation of Jenkins.getInstanceOrNull to see what you are doing wrong.
       at jenkins.model.Jenkins.get(Jenkins.java:758)
       at hudson.init.impl.InstallUncaughtExceptionHandler.lambda$init$0(InstallUncaughtExceptionHandler.java:36)
       at org.kohsuke.stapler.compression.CompressionFilter.reportException(CompressionFilter.java:77)
       at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:58)
       at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
       at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
       at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
       at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
       at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
       at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
       at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
       at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
       at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
       at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
       at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
       at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
       at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242)
       at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
       at org.eclipse.jetty.server.Server.handle(Server.java:503)
       at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
       at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
       at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
       at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
       at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
       at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
       at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
       at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
       at java.lang.Thread.run(Unknown Source)

       

        Attachments

          Activity

          Hide
          dnusbaum Devin Nusbaum added a comment - - edited

          No, but maybe if the problem is inbound HTTP requests hitting the servlet/Stapler layer while Jenkins is shutting down/restarting then blocking all HTTP traffic to Jenkins while the restart is happening might help (only really possible if you have some kind of reverse proxy in front of Jenkins or something, and I am not sure what kind of configuration would be needed).

          Edit: Manuel Cardenas thinking about it some more, I think you could also work around the issue by passing -Dhudson.model.UsageStatistics.disabled=true as a system property when starting Jenkins so that this comparison short circuits before it gets to the Jenkins.get() call.

          Show
          dnusbaum Devin Nusbaum added a comment - - edited No, but maybe if the problem is inbound HTTP requests hitting the servlet/Stapler layer while Jenkins is shutting down/restarting then blocking all HTTP traffic to Jenkins while the restart is happening might help (only really possible if you have some kind of reverse proxy in front of Jenkins or something, and I am not sure what kind of configuration would be needed). Edit: Manuel Cardenas thinking about it some more, I think you could also work around the issue by passing -Dhudson.model.UsageStatistics.disabled=true as a system property when starting Jenkins so that this comparison short circuits before it gets to the Jenkins.get() call.
          Hide
          mcardenasblanco Manuel Cardenas added a comment -

          Devin Nusbaum are you aware of any potential workaround for this?

          Show
          mcardenasblanco Manuel Cardenas added a comment - Devin Nusbaum are you aware of any potential workaround for this?
          Hide
          dnusbaum Devin Nusbaum added a comment -

          I think the problem in Manuel Cardenas's case would be fixed by https://github.com/jenkinsci/jenkins/pull/3712 which made it into the 2.150 LTS line but not 2.138. Even so, it's not clear to me why that issue would happen consistently during a restart.

          Show
          dnusbaum Devin Nusbaum added a comment - I think the problem in Manuel Cardenas 's case would be fixed by  https://github.com/jenkinsci/jenkins/pull/3712  which made it into the 2.150 LTS line but not 2.138. Even so, it's not clear to me why that issue would happen consistently during a restart.
          Hide
          mcardenasblanco Manuel Cardenas added a comment -

          Thank you for your analysis Wadeck Follonier, we will try to dig a little bit more on what is happening as it is preventing us from applying a security upgrade.

          Show
          mcardenasblanco Manuel Cardenas added a comment - Thank you for your analysis Wadeck Follonier , we will try to dig a little bit more on what is happening as it is preventing us from applying a security upgrade.
          Hide
          wfollonier Wadeck Follonier added a comment -

          Manuel Cardenas From my PoV, you are not affected by the same exception (rememberMe) but another introduced in previous release. It seems very likely that the code in rememberMe is not the culprit.

          In your case the full stacktrace shows:

          Caused by: java.lang.IllegalStateException: Jenkins.instance is missing. Read the documentation of Jenkins.getInstanceOrNull to see what you are doing wrong.
          at jenkins.model.Jenkins.get(Jenkins.java:758)
          at jenkins.telemetry.impl.StaplerDispatches$StaplerTrace.record(StaplerDispatches.java:105)
          at org.kohsuke.stapler.EvaluationTrace$ApplicationTracer.trace(EvaluationTrace.java:77)
          at org.kohsuke.stapler.Dispatcher.anonymizedTraceEval(Dispatcher.java:76)
          at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:685)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:668)
          at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
          at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
          at com.cloudbees.jenkins.ha.HAHealthCheckFilter.doFilter(HAHealthCheckFilter.java:35)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
          at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
          at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
          at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:64)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
          at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:169)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
          at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
          at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
          at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
          at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
          at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
          at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
          at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
          at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
          at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
          at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
          at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
          at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
          at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
          at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
          at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:56)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
          at org.eclipse.jetty.server.Server.handle(Server.java:531)
          at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
          ... 12 more
          

          So I created 2 PR for this ticket, 3805 to correct the rememberMe but I highly suspect to not correct the problem and 3806 to correct the legacy behavior there.
          From my PoV, the root cause is something else, in the startup mechanism that caused both exceptions.

          Show
          wfollonier Wadeck Follonier added a comment - Manuel Cardenas From my PoV, you are not affected by the same exception (rememberMe) but another introduced in previous release. It seems very likely that the code in rememberMe is not the culprit. In your case the full stacktrace shows: Caused by: java.lang.IllegalStateException: Jenkins.instance is missing. Read the documentation of Jenkins.getInstanceOrNull to see what you are doing wrong. at jenkins.model.Jenkins.get(Jenkins.java:758) at jenkins.telemetry.impl.StaplerDispatches$StaplerTrace.record(StaplerDispatches.java:105) at org.kohsuke.stapler.EvaluationTrace$ApplicationTracer.trace(EvaluationTrace.java:77) at org.kohsuke.stapler.Dispatcher.anonymizedTraceEval(Dispatcher.java:76) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:685) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:668) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655) at com.cloudbees.jenkins.ha.HAHealthCheckFilter.doFilter(HAHealthCheckFilter.java:35) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:64) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:169) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:56) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:531) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352) ... 12 more So I created 2 PR for this ticket, 3805 to correct the rememberMe but I highly suspect to not correct the problem and 3806 to correct the legacy behavior there. From my PoV, the root cause is something else, in the startup mechanism that caused both exceptions.

            People

            • Assignee:
              wfollonier Wadeck Follonier
              Reporter:
              daniel15 Daniel Lo Nigro
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated: