Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-59992

"Jenkins.instance is missing" error when restarting Jenkins

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Component/s: core
    • Labels:
      None
    • Environment:
    • Similar Issues:

      Description

      Auto-Upgrade Jenkins, select Restart Jenkins when complete...

      Manual restart via https://<jenkins>/restart...

       

      Logs (see below) show shutdown and restart occurs. However, during startup, an infinite cycle of exceptions (Jenkins.instance is missing) see below. Must manually restart windows service to successfully restart jenkins.

       

      2019-10-30 19:58:18.332+0000 [id=154905] INFO h.model.UpdateCenter$DownloadJob#run: Starting the installation of jenkins.war on behalf of xxxxx
      2019-10-30 19:58:18.332+0000 [id=154905] INFO h.model.UpdateCenter$DownloadJob#run: Starting the installation of jenkins.war on behalf of xxxxx
      2019-10-30 19:58:18.532+0000 [id=154905] INFO h.m.UpdateCenter$UpdateCenterConfiguration#download: Downloading jenkins.war
      2019-10-30 19:58:23.497+0000 [id=154905] INFO h.model.UpdateCenter$DownloadJob#run: Installation successful: jenkins.war
      2019-10-30 20:01:07.466+0000 [id=151608] INFO hudson.model.UpdateCenter#doSafeRestart: Scheduling Jenkins reboot
      2019-10-30 20:01:07.478+0000 [id=155008] INFO jenkins.model.Jenkins$23#run: Restart in 10 seconds
      2019-10-30 20:01:17.479+0000 [id=155008] INFO jenkins.model.Jenkins$23#run: Restarting VM as requested by xxxxx
      2019-10-30 20:01:17.481+0000 [id=155008] INFO jenkins.model.Jenkins#cleanUp: Stopping Jenkins
      2019-10-30 20:01:17.529+0000 [id=155008] INFO jenkins.model.Jenkins$18#onAttained: Started termination
      2019-10-30 20:01:17.542+0000 [id=155008] INFO jenkins.model.Jenkins$18#onAttained: Completed termination
      2019-10-30 20:01:17.542+0000 [id=155008] INFO jenkins.model.Jenkins#_cleanUpDisconnectComputers: Starting node disconnection
      2019-10-30 20:01:17.550+0000 [id=155008] INFO j.s.DefaultJnlpSlaveReceiver#channelClosed: safe-restart thread for xxxx terminated: java.nio.channels.ClosedChannelException
      2019-10-30 20:01:17.557+0000 [id=155008] INFO j.s.DefaultJnlpSlaveReceiver#channelClosed: safe-restart thread for xxxx terminated: java.nio.channels.ClosedChannelException
      2019-10-30 20:01:18.569+0000 [id=155008] INFO jenkins.model.Jenkins#_cleanUpShutdownPluginManager: Stopping plugin manager
      2019-10-30 20:01:18.596+0000 [id=155008] INFO jenkins.model.Jenkins#_cleanUpPersistQueue: Persisting build queue
      2019-10-30 20:01:18.602+0000 [id=155008] INFO jenkins.model.Jenkins#_cleanUpAwaitDisconnects: Waiting for node disconnection completion
      2019-10-30 20:01:18.602+0000 [id=155008] INFO jenkins.model.Jenkins#cleanUp: Jenkins stopped
      2019-10-30 20:01:22.820+0000 [id=151613] WARNING h.s.HttpSessionContextIntegrationFilter2#hasInvalidSessionSeed: Encountered IllegalStateException trying to get a user. System init may not have completed yet. Invalidating user session.
      2019-10-30 20:01:27.573+0000 [id=152096] WARNING o.e.jetty.server.HttpChannel#handleException: /tcpSlaveAgentListener/java.lang.IllegalStateException: Jenkins.instance is missing. Read the documentation of Jenkins.getInstanceOrNull to see what you are doing wrong.
      	at jenkins.model.Jenkins.get(Jenkins.java:777)
      	at org.jenkinsci.plugins.matrixauth.AuthorizationContainer.hasPermission(AuthorizationContainer.java:169)
      	at hudson.security.GlobalMatrixAuthorizationStrategy$AclImpl.hasPermission(GlobalMatrixAuthorizationStrategy.java:127)
      	at hudson.security.SidACL._hasPermission(SidACL.java:70)
      	at hudson.security.SidACL.hasPermission(SidACL.java:52)
      	at hudson.security.ACL.checkPermission(ACL.java:72)
      	at hudson.security.AccessControlled.checkPermission(AccessControlled.java:47)
      	at jenkins.model.Jenkins.getTarget(Jenkins.java:4763)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:703)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:676)
      	at hudson.init.impl.InstallUncaughtExceptionHandler.lambda$init$0(InstallUncaughtExceptionHandler.java:36)
      	at org.kohsuke.stapler.compression.CompressionFilter.reportException(CompressionFilter.java:77)
      	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:58)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
      	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
      	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1700)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1667)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      	at org.eclipse.jetty.server.Server.handle(Server.java:505)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
      	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
      	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
      	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
      	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
      	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
      	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
      	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
      	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804)
      	at java.lang.Thread.run(Unknown Source)
      

       

       

        Attachments

          Activity

          Hide
          wfollonier Wadeck Follonier added a comment - - edited

          Thank you Craig Lutgen for the report. Due to the stack trace, the symptoms are different from JENKINS-55070. Here the problem is revealed in AuthorizationContainer.java#L169.

          After a deeper investigation, the problem seems to be after the nullification of theInstance in Jenkins#L3400. There is a moment between the instance being null and the real restart (fraction of second) and that that time, the instance cannot be retrieved.

          As the HudsonIsRestarting is only used as the last filter, it has no chance to avoid other computation yet. The problem is when a filter that is applied earlier, requires to have access to Jenkins.get().

          I see two opportunities to correct the problem:

          1. Ensure every filter or class being potentially used by a filter, to not use Jenkins.get(), but instead Jenkins.getInstanceOrNull() and act correctly in case it's null. This could require a lots of work and hard time explaining the situation to people.
          2. Prevent the regular filters to be applied when the HudsonIsRestarting is the "app" in the servletContext.

          To elaborate on option 2, current flow of filters to servlet:

          • DiagnosticThreadNameFilter
          • CharacterEncodingFilter
          • CompressionFilter
          • HudsonFilter
            • ChainedServletFilter
              • HttpSessionContextIntegrationFilter2
              • BasicHeaderProcessor
              • AuthenticationProcessingFilter2
              • RememberMeProcessingFilter
              • AnonymousProcessingFilter
              • ExceptionTranslationFilter
              • UnwrapSecurityExceptionFilter
          • CrumbFilter
          • PluginServletFilter
            • UserLanguages
            • [...] << PluginServletFilter.addFilter (mainly for legacy plugins)
          • Stapler as the main Servlet

          We could either bypass completely the filters (HudsonFilter + CrumbFilter + PluginServletFilter) and reach directly the app (Stapler), or disabling the configured filters. If we bypass the rest of the filter, we will have troubles to deliver the asset bounded to the restart page, but this could be done "inline" and avoid any other parallel call.

          That could solve the
          > WARNING h.s.HttpSessionContextIntegrationFilter2#hasInvalidSessionSeed: Encountered IllegalStateException trying to get a user. System init may not have completed yet. Invalidating user session.

          But I was not able to reproduce the second warning, as I was not using the WindowsService. I imagine it's somewhat linked.

          Show
          wfollonier Wadeck Follonier added a comment - - edited Thank you Craig Lutgen for the report. Due to the stack trace, the symptoms are different from JENKINS-55070 . Here the problem is revealed in AuthorizationContainer.java#L169 . After a deeper investigation, the problem seems to be after the nullification of theInstance in Jenkins#L3400 . There is a moment between the instance being null and the real restart (fraction of second) and that that time, the instance cannot be retrieved. As the HudsonIsRestarting is only used as the last filter, it has no chance to avoid other computation yet. The problem is when a filter that is applied earlier, requires to have access to Jenkins.get() . I see two opportunities to correct the problem: Ensure every filter or class being potentially used by a filter, to not use Jenkins.get() , but instead Jenkins.getInstanceOrNull() and act correctly in case it's null. This could require a lots of work and hard time explaining the situation to people. Prevent the regular filters to be applied when the HudsonIsRestarting is the "app" in the servletContext. To elaborate on option 2, current flow of filters to servlet: DiagnosticThreadNameFilter CharacterEncodingFilter CompressionFilter HudsonFilter ChainedServletFilter HttpSessionContextIntegrationFilter2 BasicHeaderProcessor AuthenticationProcessingFilter2 RememberMeProcessingFilter AnonymousProcessingFilter ExceptionTranslationFilter UnwrapSecurityExceptionFilter CrumbFilter PluginServletFilter UserLanguages [...] << PluginServletFilter.addFilter (mainly for legacy plugins) Stapler as the main Servlet We could either bypass completely the filters (HudsonFilter + CrumbFilter + PluginServletFilter) and reach directly the app (Stapler), or disabling the configured filters. If we bypass the rest of the filter, we will have troubles to deliver the asset bounded to the restart page, but this could be done "inline" and avoid any other parallel call. That could solve the > WARNING h.s.HttpSessionContextIntegrationFilter2#hasInvalidSessionSeed: Encountered IllegalStateException trying to get a user. System init may not have completed yet. Invalidating user session. But I was not able to reproduce the second warning, as I was not using the WindowsService. I imagine it's somewhat linked.

            People

            • Assignee:
              Unassigned
              Reporter:
              csclutgen Craig Lutgen
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: