Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-72804

Jenkins not starting on Tomcat due to race condition

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • core
    • None

      Since a couple of versions we are having sometimes an issue with the startup of Jenkins.

      Early in the startup process we see the following error in the catalina.out file:

      03-Mar-2024 00:13:25.684 SEVERE [main] org.apache.catalina.session.StandardManager.startInternal Exception loading sessions from persistent storage
        java.io.EOFException
                at java.base/java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:3215)
                at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1643)
                at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:499)
                at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:457)
                at org.apache.catalina.session.StandardSession.doReadObject(StandardSession.java:1571)
                at org.apache.catalina.session.StandardSession.readObjectData(StandardSession.java:1050)
                at org.apache.catalina.session.StandardManager.doLoad(StandardManager.java:219)
                at org.apache.catalina.session.StandardManager.load(StandardManager.java:162)
                at org.apache.catalina.session.StandardManager.startInternal(StandardManager.java:356)
                at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
                at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5247)
                at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
                at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:726)
                at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:698)
                at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:696)
                at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1185)
                at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1933)
                at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
                at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
                at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
                at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
                at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1095)
                at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:477)
                at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1618)
                at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:319)
                at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
                at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423)
                at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366)
                at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:946)
                at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:835)
                at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
                at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1396)
                at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1386)
                at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
                at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
                at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
                at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:919)
                at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:265)
                at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
                at org.apache.catalina.core.StandardService.startInternal(StandardService.java:432)
                at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
                at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930)
                at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
                at org.apache.catalina.startup.Catalina.start(Catalina.java:772)
                at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.base/java.lang.reflect.Method.invoke(Method.java:566)
                at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:345)
                at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:476)
03-Mar-2024 00:13:25.719 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal One or more Filters failed to start. Full details will be found in the appropriate container log file
03-Mar-2024 00:13:25.719 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal Context [] startup failed due to previous errors
03-Mar-2024 00:13:25.806 INFO [main] hudson.lifecycle.Lifecycle.onStatusUpdate Stopping Jenkins

       

      Right after that in the log of Jenkins we see the following error:

      03-Mar-2024 00:13:25.717 SEVERE [main] org.apache.catalina.core.StandardContext.filterStart Exception starting filter [authentication-filter]
        java.lang.IllegalArgumentException: key cannot be empty or null
                at org.springframework.util.Assert.hasLength(Assert.java:243)
                at org.springframework.security.web.authentication.rememberme.AbstractRememberMeServices.<init>(AbstractRememberMeServices.java:102)
                at hudson.security.TokenBasedRememberMeServices2.<init>(TokenBasedRememberMeServices2.java:95)
                at hudson.security.SecurityRealm$SecurityComponents.createRememberMeService(SecurityRealm.java:828)
                at hudson.security.SecurityRealm$SecurityComponents.<init>(SecurityRealm.java:794)
                at hudson.security.SecurityRealm$None.createSecurityComponents(SecurityRealm.java:676)
                at hudson.security.SecurityRealm.getSecurityComponents(SecurityRealm.java:556)
                at hudson.security.HudsonFilter.reset(HudsonFilter.java:138)
                at hudson.security.HudsonFilter.init(HudsonFilter.java:112)
                at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:272)
                at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:253)
                at org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:102)
                at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4613)
                at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5256)
                at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
                at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:726)
                at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:698)
                at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:696)
                at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1185)
                at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1933)
                at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
                at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
                at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
                at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
                at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1095)
                at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:477)
                at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1618)
                at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:319)
                at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
                at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423)
                at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366)
                at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:946)
                at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:835)
                at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
                at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1396)
                at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1386)
                at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
                at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
                at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
                at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:919)
                at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:265)
                at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
                at org.apache.catalina.core.StandardService.startInternal(StandardService.java:432)
                at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
                at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930)
                at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
                at org.apache.catalina.startup.Catalina.start(Catalina.java:772)
                at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.base/java.lang.reflect.Method.invoke(Method.java:566)
                at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:345)
                at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:476)

      Could the startup fail due to the session error ? Or is this some race condition in filters/services being loaded ?

      In the past when this happened we tried to restart tomcat but we had to try multiple times before Jenkins wanted to start.

      The solution we found was to touch the web.xml file to trigger a reload of the jenkins war file.

      Current version of Jenkins is 2.414 and the operating system is RHEL 7 and RHEL 8.

       

            Unassigned Unassigned
            bartdevriendt Bart Devriendt
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: