Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-39669

Builds fail when loading jenkins.util.SystemProperties caused by java.lang.NoClassDefFoundError: javax/servlet/ServletContextListener

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Blocker
    • Resolution: Duplicate
    • Component/s: core, maven-plugin
    • Labels:
    • Environment:
      Jenkins 2.19.1, 2.19.2
      java.version 1.8.0_112
      java.vm.vendor Oracle Corporation
      java.vm.version 25.112-b15
      os.arch amd64
      os.name Linux
      BUILD_TIMESTAMP 20161111_1034
      Maven 3.3.9
    • Similar Issues:

      Description

      We have a similar exception as in JENKINS-35184 but this happens with builds only on master. In the hope that it is fixed we upgraded to 2.19.2 but it didn't help.

      The exception occurs on a random basis and cannot be pinpointed to a specific build, plugin, time,..
      When it happens, all builds after that tend to fail as well. We currently have to do a restart of Jenkins on a daily basis and sometimes multiple times.

      Restarting Jenkins is the only workaround we currently have and is growing into a pain especially when we are releasing. This issue is a blocker for us.

      Our setup is:

      • one master Jenkins running mostly all builds
      • one Windows slave for running selenium tests
      • most builds are run with Maven using the Maven Project Type job (seems to be important)

      StackTrace from the build logs:

      java.io.IOException: Remote call on Channel to Maven [/opt/jdk1.8.0_112/bin/java, -cp, /opt/jenkins/home/plugins/maven-plugin/WEB-INF/lib/maven33-agent-1.8.1.jar:/opt/apache-maven-3.3.9/boot/plexus-classworlds-2.5.2.jar:/opt/apache-maven-3.3.9/conf/logging, jenkins.maven3.agent.Maven33Main, /opt/apache-maven-3.3.9, /var/cache/jenkins/war/WEB-INF/lib/remoting-2.62.jar, /opt/jenkins/home/plugins/maven-plugin/WEB-INF/lib/maven33-interceptor-1.8.1.jar, /opt/jenkins/home/plugins/maven-plugin/WEB-INF/lib/maven3-interceptor-commons-1.8.1.jar, 36506] failed
      	at hudson.remoting.Channel.call(Channel.java:805)
      	at hudson.maven.ProcessCache$MavenProcess.call(ProcessCache.java:161)
      	at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:873)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534)
      	at hudson.model.Run.execute(Run.java:1720)
      	at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:544)
      	at hudson.model.ResourceController.execute(ResourceController.java:98)
      	at hudson.model.Executor.run(Executor.java:404)
      Caused by: java.lang.LinkageError: Failed to load jenkins.util.SystemProperties
      	at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:377)
      	at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:285)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      	at hudson.Main.<clinit>(Main.java:222)
      	at hudson.EnvVars.initMaster(EnvVars.java:432)
      	at hudson.EnvVars.<clinit>(EnvVars.java:427)
      	at java.io.ObjectStreamClass.hasStaticInitializer(Native Method)
      	at java.io.ObjectStreamClass.computeDefaultSUID(ObjectStreamClass.java:1787)
      	at java.io.ObjectStreamClass.access$100(ObjectStreamClass.java:72)
      	at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:253)
      	at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:251)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at java.io.ObjectStreamClass.getSerialVersionUID(ObjectStreamClass.java:250)
      	at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:611)
      	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1630)
      	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
      	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
      	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
      	at hudson.remoting.UserRequest.deserialize(UserRequest.java:217)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:131)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:50)
      	at hudson.remoting.Request$2.run(Request.java:332)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      	at ......remote call to Channel to Maven [/opt/jdk1.8.0_112/bin/java, -cp, /opt/jenkins/home/plugins/maven-plugin/WEB-INF/lib/maven33-agent-1.8.1.jar:/opt/apache-maven-3.3.9/boot/plexus-classworlds-2.5.2.jar:/opt/apache-maven-3.3.9/conf/logging, jenkins.maven3.agent.Maven33Main, /opt/apache-maven-3.3.9, /var/cache/jenkins/war/WEB-INF/lib/remoting-2.62.jar, /opt/jenkins/home/plugins/maven-plugin/WEB-INF/lib/maven33-interceptor-1.8.1.jar, /opt/jenkins/home/plugins/maven-plugin/WEB-INF/lib/maven3-interceptor-commons-1.8.1.jar, 36506](Native Method)
      	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1433)
      	at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
      	at hudson.remoting.Channel.call(Channel.java:797)
      	... 7 more
      Caused by: java.lang.NoClassDefFoundError: javax/servlet/ServletContextListener
      	at java.lang.ClassLoader.defineClass1(Native Method)
      	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
      	at java.lang.ClassLoader.defineClass(ClassLoader.java:642)
      	at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:373)
      	at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:285)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      	at hudson.Main.<clinit>(Main.java:222)
      	at hudson.EnvVars.initMaster(EnvVars.java:432)
      	at hudson.EnvVars.<clinit>(EnvVars.java:427)
      	at java.io.ObjectStreamClass.hasStaticInitializer(Native Method)
      	at java.io.ObjectStreamClass.computeDefaultSUID(ObjectStreamClass.java:1787)
      	at java.io.ObjectStreamClass.access$100(ObjectStreamClass.java:72)
      	at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:253)
      	at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:251)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at java.io.ObjectStreamClass.getSerialVersionUID(ObjectStreamClass.java:250)
      	at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:611)
      	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1630)
      	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
      	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
      	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
      	at hudson.remoting.UserRequest.deserialize(UserRequest.java:217)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:131)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:50)
      	at hudson.remoting.Request$2.run(Request.java:332)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.ClassNotFoundException: javax.servlet.ServletContextListener (java.net.URLClassLoader@3f99bd52 did not find class file)
      	at hudson.remoting.RemoteClassLoader$ClassLoaderProxy.fetch2(RemoteClassLoader.java:792)
      	at hudson.remoting.RemoteClassLoader$ClassLoaderProxy.fetch4(RemoteClassLoader.java:842)
      	at hudson.remoting.RemoteClassLoader$ClassLoaderProxy.fetch3(RemoteClassLoader.java:849)
      	at sun.reflect.GeneratedMethodAccessor208.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:887)
      	at hudson.remoting.Request$2.run(Request.java:332)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
      	at org.jenkinsci.remoting.CallableDecorator.call(CallableDecorator.java:18)
      	at hudson.remoting.CallableDecoratorList$1.call(CallableDecoratorList.java:21)
      	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      	at ......remote call to channel(Native Method)
      	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1433)
      	at hudson.remoting.Request.call(Request.java:172)
      	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:255)
      	at com.sun.proxy.$Proxy2.fetch3(Unknown Source)
      	at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:187)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      	... 36 more
      

      Installed plugins

      ace-editor	1.1	true	false
      active-directory	1.49	true	false
      analysis-core	1.79	true	false
      android-emulator	2.15	true	false
      android-lint	2.4	true	false
      ant	1.4	true	false
      antisamy-markup-formatter	1.5	true	false
      authorize-project	1.2.2	true	false
      bouncycastle-api	2.16.0	true	false
      branch-api	1.11	true	false
      build-failure-analyzer	1.17.2	true	false
      build-monitor-plugin	1.10+build.201610041454	true	false
      build-name-setter	1.6.5	true	false
      build-pipeline-plugin	1.5.4	true	false
      build-timeout	1.17.1	true	false
      build-timestamp	1.0.1	true	false
      claim	2.8	true	false
      cloudbees-folder	5.13	true	false
      conditional-buildstep	1.3.5	true	false
      config-file-provider	2.13	true	false
      credentials	2.1.8	true	false
      credentials-binding	1.9	true	false
      dashboard-view	2.9.10	true	false
      display-url-api	0.5	true	false
      durable-task	1.12	true	false
      external-monitor-job	1.6	true	false
      git	3.0.0	true	false
      git-client	2.0.0	true	false
      git-parameter	0.7.0	true	false
      git-server	1.7	true	false
      gradle	1.25	true	false
      greenballs	1.15	true	false
      handlebars	1.1.1	true	false
      htmlpublisher	1.11	true	false
      icon-shim	2.0.3	true	false
      javadoc	1.4	true	false
      jobConfigHistory	2.15	true	false
      jobgenerator	1.22	true	false
      jquery	1.11.2-0	true	false
      jquery-detached	1.2.1	true	false
      junit	1.19	true	false
      ldap	1.13	true	false
      locale	1.2	true	false
      log-parser	2.0	true	false
      m2release	0.14.0	true	false
      mailer	1.18	true	false
      matrix-auth	1.4	true	false
      matrix-project	1.7.1	true	false
      maven-plugin	2.14	true	false
      momentjs	1.1.1	true	false
      nodejs	0.2.1	true	false
      pam-auth	1.3	true	false
      parameterized-trigger	2.32	true	false
      pipeline-build-step	2.3	true	false
      pipeline-graph-analysis	1.2	true	false
      pipeline-input-step	2.3	true	false
      pipeline-milestone-step	1.1	true	false
      pipeline-rest-api	2.1	true	false
      pipeline-stage-step	2.2	true	false
      pipeline-stage-view	2.1	true	false
      plain-credentials	1.3	true	false
      port-allocator	1.8	true	false
      protex-jenkins	1.4.0	true	false
      publish-over-ssh	1.14	true	false
      run-condition	1.0	true	false
      scm-api	1.3	true	false
      script-security	1.24	true	false
      scriptler	2.9	true	false
      sectioned-view	1.20	true	false
      sonar	2.5	true	false
      ssh-agent	1.13	true	false
      ssh-credentials	1.12	true	false
      ssh-slaves	1.11	true	false
      stash-pullrequest-builder	1.7.0	true	false
      stashNotifier	1.11.4	true	false
      structs	1.5	true	false
      thinBackup	1.7.4	true	false
      token-macro	2.0	true	false
      windows-slaves	1.2	true	false
      workflow-aggregator	2.4	true	false
      workflow-api	2.5	true	false
      workflow-basic-steps	2.2	true	false
      workflow-cps	2.21	true	false
      workflow-cps-global-lib	2.4	true	false
      workflow-durable-task-step	2.5	true	false
      workflow-job	2.8	true	false
      workflow-multibranch	2.9	true	false
      workflow-scm-step	2.2	true	false
      workflow-step-api	2.4	true	false
      workflow-support	2.10	true	false
      ws-cleanup	0.30	true	false
      

      We added additional loggers: hudson.remoting.RemoteClassLoader and org.jenkinsci.plugins but they did not provide helpful information.

      As discussed in JENKINS-35184 this might be caused by a plugin especially the maven plugin thus I set it as component as well.

      If there is more information needed please state so, we will gladly help to get this fixed

        Attachments

          Issue Links

            Activity

            Hide
            robert Robert Klaus added a comment -

            We found a workaround - which is not nice.

            The workaround is to put the servlet-api jar (containing ServletContextListener.class) on the classpath of a Maven build. We did this by copying the jar (http://search.maven.org/remotecontent?filepath=javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar) into the /lib/ext folder of the JRE.
            This is not nice but I wanted to document it here in case somebody else runs into this problem.

            Some explanations:

            It looks like the exception stems from a remote process and we suspect it comes from Maven builds (Jobs with Maven Project Type). As these run in their own Java Process it looks like the Systemproperties are somewhat loaded in that process but don't have the necessary dependencies.
            After looking in the pom.xml on Github, we saw the dependency to the javax.servlet-api in scope "provided".
            The Jar itself is contained in the winstone container but not in standard Java libraries.
            As cannot figure out when this happens it was worth a try to just provide the class to see if it works or produces a different error.

            Builds are stable since two days - but still .. this bugs needs to be fixed.

            Show
            robert Robert Klaus added a comment - We found a workaround - which is not nice. The workaround is to put the servlet-api jar (containing ServletContextListener.class) on the classpath of a Maven build. We did this by copying the jar ( http://search.maven.org/remotecontent?filepath=javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar ) into the /lib/ext folder of the JRE. This is not nice but I wanted to document it here in case somebody else runs into this problem. Some explanations: It looks like the exception stems from a remote process and we suspect it comes from Maven builds (Jobs with Maven Project Type). As these run in their own Java Process it looks like the Systemproperties are somewhat loaded in that process but don't have the necessary dependencies. After looking in the pom.xml on Github, we saw the dependency to the javax.servlet-api in scope "provided". The Jar itself is contained in the winstone container but not in standard Java libraries. As cannot figure out when this happens it was worth a try to just provide the class to see if it works or produces a different error. Builds are stable since two days - but still .. this bugs needs to be fixed.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            it's just another issue with Maven interceptors. SystemProperties class in Jenkins requires ServletContextListener, and seems the the missing class causes fatal failure, because the class is not being sent over the channel due to security reasons IIRC.

            Show
            oleg_nenashev Oleg Nenashev added a comment - it's just another issue with Maven interceptors. SystemProperties class in Jenkins requires ServletContextListener, and seems the the missing class causes fatal failure, because the class is not being sent over the channel due to security reasons IIRC.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Nicolas De Loof
            Path:
            docker-plugin/pom.xml
            docker-plugin/src/main/java/com/nirima/jenkins/plugins/docker/utils/JenkinsUtils.java
            docker-plugin/src/main/java/io/jenkins/docker/pipeline/DockerNodeStep.java
            docker-plugin/src/main/java/io/jenkins/docker/pipeline/DockerNodeStepExecution.java
            docker-plugin/src/main/resources/io/jenkins/docker/pipeline/DockerNodeStep/config.jelly
            http://jenkins-ci.org/commit/docker-plugin/9bf481aac670959542dcb8f576e1e803b422b895
            Log:
            upgrade to latest Jenkins and dependencies
            workaround for resurrecting JENKINS-39669

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Nicolas De Loof Path: docker-plugin/pom.xml docker-plugin/src/main/java/com/nirima/jenkins/plugins/docker/utils/JenkinsUtils.java docker-plugin/src/main/java/io/jenkins/docker/pipeline/DockerNodeStep.java docker-plugin/src/main/java/io/jenkins/docker/pipeline/DockerNodeStepExecution.java docker-plugin/src/main/resources/io/jenkins/docker/pipeline/DockerNodeStep/config.jelly http://jenkins-ci.org/commit/docker-plugin/9bf481aac670959542dcb8f576e1e803b422b895 Log: upgrade to latest Jenkins and dependencies workaround for resurrecting JENKINS-39669
            Hide
            jglick Jesse Glick added a comment -

            I think this case was fixed in JENKINS-46386.

            Show
            jglick Jesse Glick added a comment - I think this case was fixed in JENKINS-46386 .

              People

              • Assignee:
                Unassigned
                Reporter:
                robert Robert Klaus
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: