Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-20707

NullPointerException in MultiClassLoaderSerializer when Warnings plugin processes puppet-lint output

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Reopened (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: core, remoting
    • Labels:
      None
    • Environment:
      Cloudbees managed Jenkins, warnings plugin 4.38
    • Similar Issues:

      Description

      We are running a Puppet-Lint run on Cloudbees Jenkins and processing the output with the Warnings Parser Plugin. Puppet-Lint seems to run ok, but sometimes crashes with the following NPE:

      [WARNINGS] Parsing warnings in console log with parser Puppet-Lint
      ERROR: Publisher hudson.plugins.warnings.WarningsPublisher aborted due to exception
      java.lang.NullPointerException
      at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:95)
      at hudson.remoting.MultiClassLoaderSerializer$Input.resolveClass(MultiClassLoaderSerializer.java:112)
      at java.io.ObjectInputStream.readNonProxyDesc(Unknown Source)
      at java.io.ObjectInputStream.readClassDesc(Unknown Source)
      at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
      at java.io.ObjectInputStream.readObject0(Unknown Source)
      at java.io.ObjectInputStream.readObject(Unknown Source)
      at java.util.HashSet.readObject(Unknown Source)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        Attachments

          Activity

          Hide
          kohsuke Kohsuke Kawaguchi added a comment -

          ClassLoader is always exported with autoUnexport=false (MultiClassLoaderSerializer.Output.annotateClass), so its unexporting is subject to the garbage collection on the other side. When RemoteClassLoader gets GC-ed, its proxy is RemoteIClassLoader and that in turn points to a generated proxy that wraps RemoteInvocationHandler. The finalizer of this object sends an unexport command back to the other side.

          Therefore indeed classloaders are getting exported and unexported pretty rapidly, which seems rather wasteful. It forces all the class loading to be redone, for one thing. There's a method to prevent exported objects from getting unexported (Channel.pin and Channel.pinClassLoader), but this is only used once in the Jenkins core to pin down the Jenkins core classloader (SlaveComputer.setChannel) and not plugins. Given that Jenkins doesn't unload plugins, I think we should just pin down all the plugin classloaders as well.

          So far, I think my hypothesis that I mentioned in the previous comment is indeed possible. I'm going to write a test case to reproduce this.

          Show
          kohsuke Kohsuke Kawaguchi added a comment - ClassLoader is always exported with autoUnexport=false (MultiClassLoaderSerializer.Output.annotateClass), so its unexporting is subject to the garbage collection on the other side. When RemoteClassLoader gets GC-ed, its proxy is RemoteIClassLoader and that in turn points to a generated proxy that wraps RemoteInvocationHandler. The finalizer of this object sends an unexport command back to the other side. Therefore indeed classloaders are getting exported and unexported pretty rapidly, which seems rather wasteful. It forces all the class loading to be redone, for one thing. There's a method to prevent exported objects from getting unexported (Channel.pin and Channel.pinClassLoader), but this is only used once in the Jenkins core to pin down the Jenkins core classloader (SlaveComputer.setChannel) and not plugins. Given that Jenkins doesn't unload plugins, I think we should just pin down all the plugin classloaders as well. So far, I think my hypothesis that I mentioned in the previous comment is indeed possible. I'm going to write a test case to reproduce this.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          src/test/java/hudson/remoting/Bug20707Test.java
          src/test/java/hudson/remoting/ClassRemotingTest.java
          src/test/java/hudson/remoting/DummyClassLoader.java
          src/test/java/hudson/remoting/DummyClassLoaderTest.java
          src/test/java/hudson/remoting/TestEcho.java
          src/test/java/hudson/remoting/util/OneShotEvent.java
          http://jenkins-ci.org/commit/remoting/bd8e65cd874b863fc7d45e601da9a9267478d192
          Log:
          JENKINS-20707

          I tried to write a test case to verify the hypothesis along the line of
          https://issues.jenkins-ci.org/browse/JENKINS-20707?focusedCommentId=198755&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-198755,
          but ultimately it was a dead-end.

          The reason I was wrong is because ImportedClassLoaderTable maintain strong
          references to all the imported classloaders, and it prevents them from
          getting garbage collected. Probably for the reasons I was mentioning
          in JENKINS-20707, RemoteClassLoaders never get garbage collected, so they should
          never get unexported. I've spent much time trying to make the test case
          work, so I'm going to commit it on the side just in case someone finds
          useful in the future.

          Compare: https://github.com/jenkinsci/remoting/compare/663dae9c83be^...bd8e65cd874b

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: src/test/java/hudson/remoting/Bug20707Test.java src/test/java/hudson/remoting/ClassRemotingTest.java src/test/java/hudson/remoting/DummyClassLoader.java src/test/java/hudson/remoting/DummyClassLoaderTest.java src/test/java/hudson/remoting/TestEcho.java src/test/java/hudson/remoting/util/OneShotEvent.java http://jenkins-ci.org/commit/remoting/bd8e65cd874b863fc7d45e601da9a9267478d192 Log: JENKINS-20707 I tried to write a test case to verify the hypothesis along the line of https://issues.jenkins-ci.org/browse/JENKINS-20707?focusedCommentId=198755&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-198755 , but ultimately it was a dead-end. The reason I was wrong is because ImportedClassLoaderTable maintain strong references to all the imported classloaders, and it prevents them from getting garbage collected. Probably for the reasons I was mentioning in JENKINS-20707 , RemoteClassLoaders never get garbage collected, so they should never get unexported. I've spent much time trying to make the test case work, so I'm going to commit it on the side just in case someone finds useful in the future. Compare: https://github.com/jenkinsci/remoting/compare/663dae9c83be ^...bd8e65cd874b
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          src/main/java/hudson/remoting/Channel.java
          src/main/java/hudson/remoting/ExportTable.java
          src/main/java/hudson/remoting/RemoteInvocationHandler.java
          http://jenkins-ci.org/commit/remoting/c9d61a335d8a75ef9908b80eac16a0269913de3e
          Log:
          JENKINS-20707

          Treating this error as fatal and providing basic diagnostic information

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: src/main/java/hudson/remoting/Channel.java src/main/java/hudson/remoting/ExportTable.java src/main/java/hudson/remoting/RemoteInvocationHandler.java http://jenkins-ci.org/commit/remoting/c9d61a335d8a75ef9908b80eac16a0269913de3e Log: JENKINS-20707 Treating this error as fatal and providing basic diagnostic information
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          src/main/java/hudson/remoting/ExportTable.java
          src/test/java/hudson/remoting/ExportTableTest.java
          http://jenkins-ci.org/commit/remoting/5db46de05db55262e75755ba47a99c4b12965d9b
          Log:
          JENKINS-20707 Adding more diagnosis.

          Record most recently unexported objects in an attempt to understand the problem better

          Compare: https://github.com/jenkinsci/remoting/compare/0ccbbc64bec5...5db46de05db5

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: src/main/java/hudson/remoting/ExportTable.java src/test/java/hudson/remoting/ExportTableTest.java http://jenkins-ci.org/commit/remoting/5db46de05db55262e75755ba47a99c4b12965d9b Log: JENKINS-20707 Adding more diagnosis. Record most recently unexported objects in an attempt to understand the problem better Compare: https://github.com/jenkinsci/remoting/compare/0ccbbc64bec5...5db46de05db5
          Hide
          kohsuke Kohsuke Kawaguchi added a comment -

          I'm adding more diagnostics toward 2.40 so that when this error happens in the field we have more information.

          Show
          kohsuke Kohsuke Kawaguchi added a comment - I'm adding more diagnostics toward 2.40 so that when this error happens in the field we have more information.

            People

            • Assignee:
              kohsuke Kohsuke Kawaguchi
              Reporter:
              jpollak jpollak
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: