Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-41852

Remoting object pinning can result in an object that is released immediately

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      The current logic for reference counting and pinning contains a potential bug that could give rise to objects being unexported.

              void pin() {
                  if (referenceCount<Integer.MAX_VALUE/2)
                      referenceCount += Integer.MAX_VALUE/2;
              }
      

      So if we have the sequence:

      • Export table entry created -> 0x00000000
      • Object pinned -> 0x3FFFFFFF
      • ...
      • Object released -> 0x3FFFFFFE
      • Object pinned -> 0x7FFFFFFD
      • Object addRef -> 0x7FFFFFFE
      • Object addRef -> 0x80000000
      • Object deallocated as refcount < 0

      The number one symptom of this bug is:

      2017-02-07 11:11:21.087-0500 [id=47743]	WARNING	h.n.AbstractAsyncNodeMonitorDescriptor#monitor: Failed to monitor ___REDACTED___ for Clock Differencejava.util.concurrent.ExecutionException: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(-1) in the channel exported table
      	at hudson.remoting.Channel$2.adapt(Channel.java:829)
      	at hudson.remoting.Channel$2.adapt(Channel.java:824)
      	at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
      	at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96)
      	at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
      Caused by: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(-1) in the channel exported table
      	at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:102)
      	at hudson.remoting.MultiClassLoaderSerializer$Input.resolveClass(MultiClassLoaderSerializer.java:123)
      	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612)
      	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
      	at hudson.remoting.UserRequest.deserialize(UserRequest.java:217)
      	at hudson.remoting.UserResponse.retrieve(UserRequest.java:250)
      	at hudson.remoting.Channel$2.adapt(Channel.java:827)
      	... 4 more
      Caused by: java.util.concurrent.ExecutionException: Invalid object ID 2 iota=26
      	at hudson.remoting.ExportTable.diagnoseInvalidObjectId(ExportTable.java:408)
      	at hudson.remoting.ExportTable.get(ExportTable.java:333)
      	at hudson.remoting.Channel.getExportedObject(Channel.java:633)
      	at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:100)
      	... 13 more
      Caused by: java.lang.Exception: Object appears to be deallocated at lease before Tue Feb 07 11:08:06 EST 2017
      	at hudson.remoting.ExportTable.diagnoseInvalidObjectId(ExportTable.java:404)
      	... 16 more
      

      Specifically that this is Invalid object ID 2

      object ID 1 is always the Channel exported as an IChannel

      object ID 2 is always the RemoteClassLoader$ClassLoaderProxy exported as an hudson.remoting.RemoteClassLoader$IClassLoader

      There is no circumstances I can see where object ID 2 should ever be safe to remove. In fact there is code that explicitly pins the classloader

      Strangely when I look at the export tables from agents, however I see things like:

      #2 (ref.571) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.1073744046) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.456) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.516) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.517) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.513) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.1028) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.403) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.313) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.494) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.1073742439) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.447) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.316) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.314) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.316) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.1073742160) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.324) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.321) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.326) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.324) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.318) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.323) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.320) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      

      which is interesting, because we have some of these pinned (i.e. values like 1073744046, 1073742439 and 1073742160) and some unpinned...

      This is rather indicative of something strange going on

        Attachments

          Activity

          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Stephen Connolly
          Path:
          src/main/java/hudson/remoting/ExportTable.java
          http://jenkins-ci.org/commit/remoting/849e01553c4a938add869699068c2fa1e0a67ea6
          Log:
          [FIXED JENKINS-41852] Tweak pinning logic

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: src/main/java/hudson/remoting/ExportTable.java http://jenkins-ci.org/commit/remoting/849e01553c4a938add869699068c2fa1e0a67ea6 Log: [FIXED JENKINS-41852] Tweak pinning logic
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          src/main/java/hudson/remoting/ExportTable.java
          http://jenkins-ci.org/commit/remoting/eb2aa0b2c98d1f5c486d6824b4a3f4c3f61a317f
          Log:
          Merge pull request #148 from stephenc/jenkins-41852-2.x

          [FIXED JENKINS-41852] Tweak pinning logic

          Compare: https://github.com/jenkinsci/remoting/compare/059b89357397...eb2aa0b2c98d

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/hudson/remoting/ExportTable.java http://jenkins-ci.org/commit/remoting/eb2aa0b2c98d1f5c486d6824b4a3f4c3f61a317f Log: Merge pull request #148 from stephenc/jenkins-41852-2.x [FIXED JENKINS-41852] Tweak pinning logic Compare: https://github.com/jenkinsci/remoting/compare/059b89357397...eb2aa0b2c98d
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          docs/configuration.md
          src/main/java/hudson/remoting/Base64.java
          src/main/java/hudson/remoting/Channel.java
          src/main/java/hudson/remoting/ErrorPropagatingOutputStream.java
          src/main/java/hudson/remoting/ExportTable.java
          src/main/java/hudson/remoting/LocalChannel.java
          src/main/java/hudson/remoting/RemoteInvocationHandler.java
          src/main/java/hudson/remoting/VirtualChannel.java
          http://jenkins-ci.org/commit/remoting/27bb51dd14db44952348539d15ee004c90c248b2
          Log:
          Document and annotate object Export/Unexport logic

          During last releases we have fixed several issues related to the object export/unexport operations (JENKINS-23271, JENKINS-41852). I decided that it makes sense to improve the method documentation just to make Javadocs more explicit.

          The change also adds missing documentation of hudson.remoting.ExportTable.unexportLogSize

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: docs/configuration.md src/main/java/hudson/remoting/Base64.java src/main/java/hudson/remoting/Channel.java src/main/java/hudson/remoting/ErrorPropagatingOutputStream.java src/main/java/hudson/remoting/ExportTable.java src/main/java/hudson/remoting/LocalChannel.java src/main/java/hudson/remoting/RemoteInvocationHandler.java src/main/java/hudson/remoting/VirtualChannel.java http://jenkins-ci.org/commit/remoting/27bb51dd14db44952348539d15ee004c90c248b2 Log: Document and annotate object Export/Unexport logic During last releases we have fixed several issues related to the object export/unexport operations ( JENKINS-23271 , JENKINS-41852 ). I decided that it makes sense to improve the method documentation just to make Javadocs more explicit. The change also adds missing documentation of hudson.remoting.ExportTable.unexportLogSize
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          pom.xml
          http://jenkins-ci.org/commit/jenkins/815da8aa732baa699481828dda67dd5835ba4992
          Log:
          Update remoting to 3.5

          https://github.com/jenkinsci/remoting/edit/master/CHANGELOG.md

          Fixed issues:

          Improvements:

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: pom.xml http://jenkins-ci.org/commit/jenkins/815da8aa732baa699481828dda67dd5835ba4992 Log: Update remoting to 3.5 https://github.com/jenkinsci/remoting/edit/master/CHANGELOG.md Fixed issues: JENKINS-40710 ( https://issues.jenkins-ci.org/browse/JENKINS-40710 ) - Match headers case-insensitively in `JnlpAgentEndpointResolver` in order to be compliant with HTTP2 lower-case headers. ( PR #139 ( https://github.com/jenkinsci/remoting/pull/139 ), PR #140 ( https://github.com/jenkinsci/remoting/pull/140 )) JENKINS-41513 ( https://issues.jenkins-ci.org/browse/JENKINS-41513 ) - Prevent `NullPointerException` in `JnlpAgentEndpointResolver` when receiving a header with `null` name. ( PR #140 ( https://github.com/jenkinsci/remoting/pull/140 )) JENKINS-41852 ( https://issues.jenkins-ci.org/browse/JENKINS-41852 ) - Fix exported object pinning logic to prevent release due to the integer overflow. ( PR #148 ( https://github.com/jenkinsci/remoting/pull/148 )) Improvements: JENKINS-41730 ( https://issues.jenkins-ci.org/browse/JENKINS-41730 ) - Add the new `org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.ignoreJenkinsAgentProtocolsHeader` property, which allows specifying a custom list of supported protocols instead of the one returned by the Jenkins master. ( PR #146 ( https://github.com/jenkinsci/remoting/pull/146 )) Print the Filesystem Jar Cache directory location in the error message when this cache directory is not writable. ( PR #143 ( https://github.com/jenkinsci/remoting/pull/143 )) Replace `MimicException` with the older `ProxyException` when serializing non-serializable exceptions thrown by the remote code. ( PR #141 ( https://github.com/jenkinsci/remoting/pull/141 )) Use OID of the `ClassLoaderProxy` in error message when the proxy cannot be located in the export table. ( PR #147 ( https://github.com/jenkinsci/remoting/pull/147 ))
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          pom.xml
          http://jenkins-ci.org/commit/jenkins/1763e6400f56da0300a44c69b9c10bcf9d2cafe9
          Log:
          Merge pull request #2751 from oleg-nenashev/remoting/3.5

          [JENKINS-40710, JENKINS-41513, JENKINS-41852, JENKINS-41730] - Update remoting to 3.5

          Compare: https://github.com/jenkinsci/jenkins/compare/def0f0d11266...1763e6400f56

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: pom.xml http://jenkins-ci.org/commit/jenkins/1763e6400f56da0300a44c69b9c10bcf9d2cafe9 Log: Merge pull request #2751 from oleg-nenashev/remoting/3.5 [JENKINS-40710, JENKINS-41513, JENKINS-41852, JENKINS-41730] - Update remoting to 3.5 Compare: https://github.com/jenkinsci/jenkins/compare/def0f0d11266...1763e6400f56
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          pom.xml
          http://jenkins-ci.org/commit/jenkins/08c5da35d3fafd5cf90143181e07f0d22f4f5d60
          Log:
          Update remoting to 3.5

          https://github.com/jenkinsci/remoting/edit/master/CHANGELOG.md

          Fixed issues:

          Improvements:

          (cherry picked from commit 815da8aa732baa699481828dda67dd5835ba4992)

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: pom.xml http://jenkins-ci.org/commit/jenkins/08c5da35d3fafd5cf90143181e07f0d22f4f5d60 Log: Update remoting to 3.5 https://github.com/jenkinsci/remoting/edit/master/CHANGELOG.md Fixed issues: JENKINS-40710 ( https://issues.jenkins-ci.org/browse/JENKINS-40710 ) - Match headers case-insensitively in `JnlpAgentEndpointResolver` in order to be compliant with HTTP2 lower-case headers. ( PR #139 ( https://github.com/jenkinsci/remoting/pull/139 ), PR #140 ( https://github.com/jenkinsci/remoting/pull/140 )) JENKINS-41513 ( https://issues.jenkins-ci.org/browse/JENKINS-41513 ) - Prevent `NullPointerException` in `JnlpAgentEndpointResolver` when receiving a header with `null` name. ( PR #140 ( https://github.com/jenkinsci/remoting/pull/140 )) JENKINS-41852 ( https://issues.jenkins-ci.org/browse/JENKINS-41852 ) - Fix exported object pinning logic to prevent release due to the integer overflow. ( PR #148 ( https://github.com/jenkinsci/remoting/pull/148 )) Improvements: JENKINS-41730 ( https://issues.jenkins-ci.org/browse/JENKINS-41730 ) - Add the new `org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.ignoreJenkinsAgentProtocolsHeader` property, which allows specifying a custom list of supported protocols instead of the one returned by the Jenkins master. ( PR #146 ( https://github.com/jenkinsci/remoting/pull/146 )) Print the Filesystem Jar Cache directory location in the error message when this cache directory is not writable. ( PR #143 ( https://github.com/jenkinsci/remoting/pull/143 )) Replace `MimicException` with the older `ProxyException` when serializing non-serializable exceptions thrown by the remote code. ( PR #141 ( https://github.com/jenkinsci/remoting/pull/141 )) Use OID of the `ClassLoaderProxy` in error message when the proxy cannot be located in the export table. ( PR #147 ( https://github.com/jenkinsci/remoting/pull/147 )) (cherry picked from commit 815da8aa732baa699481828dda67dd5835ba4992)

            People

            • Assignee:
              stephenconnolly Stephen Connolly
              Reporter:
              stephenconnolly Stephen Connolly
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: