Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49027

Improve diagnostics of rejected objects in Remoting

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Currently it is hard to understand which field has been rejected from serialization in Remoting. You get only stacktraces like this one:

      [android-lint] Collecting Android Lint files...
      ERROR: Build step failed with exception
      java.lang.SecurityException: Rejected: com.google.common.collect.AbstractMultimap
      	at hudson.remoting.ClassFilter.check(ClassFilter.java:75)
      	at hudson.remoting.MultiClassLoaderSerializer$Input.resolveClass(MultiClassLoaderSerializer.java:129)
      	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1613)
      	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
      	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1623)
      	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
      	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1623)
      	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
      	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
      	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
      	at hudson.remoting.UserRequest.deserialize(UserRequest.java:277)
      	at hudson.remoting.UserResponse.retrieve(UserRequest.java:310)
      	at hudson.remoting.Channel.call(Channel.java:909)
      	at hudson.FilePath.act(FilePath.java:998)
      

      I propose to...

      • If FINE/INFO logger is enabled for the class filter, run deep object analysis and log info about structure and rejected classes
      • Since there is no way to hook on Java deserialization, use Reflection for such operation
      • Nice2have: Print information about rejected classes into a separate file if workdir is enabled

        Attachments

          Issue Links

            Activity

            Hide
            jglick Jesse Glick added a comment -

            I have a functional test which reproduced the above stack trace, so I may spend a little time experimenting with a fix for this.

            Show
            jglick Jesse Glick added a comment - I have a functional test which reproduced the above stack trace, so I may spend a little time experimenting with a fix for this.
            Hide
            jglick Jesse Glick added a comment -

            Unnecessary to patch XStream, since that already gives a meaningful stack trace with lines such as

            java.lang.RuntimeException: Failed to serialize com.amazonaws.AmazonServiceException#httpHeaders for class com.amazonaws.services.s3.model.AmazonS3Exception
            

            (Observable by commenting out the agent creation in S3UploadStepIntegrationTest for JENKINS-49025, so that the S3 calls happen on the master without Remoting; then the error appears when org.jenkinsci.plugins.workflow.actions.ErrorAction attempts to save the stack trace. That is not fatal.)

            Show
            jglick Jesse Glick added a comment - Unnecessary to patch XStream, since that already gives a meaningful stack trace with lines such as java.lang.RuntimeException: Failed to serialize com.amazonaws.AmazonServiceException#httpHeaders for class com.amazonaws.services.s3.model.AmazonS3Exception (Observable by commenting out the agent creation in S3UploadStepIntegrationTest for JENKINS-49025 , so that the S3 calls happen on the master without Remoting; then the error appears when org.jenkinsci.plugins.workflow.actions.ErrorAction attempts to save the stack trace. That is not fatal.)
            Hide
            jglick Jesse Glick added a comment -

            …though I filed workflow-api PR 64 to behave better in that respect too.

            Show
            jglick Jesse Glick added a comment - …though I filed  workflow-api PR 64 to behave better in that respect too.
            Hide
            jglick Jesse Glick added a comment -

            To your suggestions:

            run deep object analysis and log info about structure

            I cannot think of any easy options here. JBoss Marshalling has better diagnostics, but uses an incompatible stream format, so it is useless here. Long ago I wrote a tool to dump Java serialization data (purely structural—without doing any actual classloading), which could be revived and brought up to Java 8, but implementing the complete Object Serialization Stream Protocol is quite tedious.

            (Also to use such a tool you would need to retain a copy of the InputStream being passed to ObjectInputStream for purposes of retroactive diagnosis, which adds some overhead.)

            use Reflection for such operation

            I spent some time playing with MultiClassLoaderSerializer.Input but found no way to obtain better diagnostics, even with reflection. There is simply no useful context available outside stack-allocated variables when resolveClass is called, and no other overridable methods which would yield anything of interest.

            Print information about rejected classes into a separate file

            Seems pointless. We already log all rejected classes. That is the most easily available information already.

            if workdir is enabled

            Remoting workdir is inapplicable here, since we are talking about the master.

            Show
            jglick Jesse Glick added a comment - To your suggestions: run deep object analysis and log info about structure I cannot think of any easy options here. JBoss Marshalling has better diagnostics, but uses an incompatible stream format, so it is useless here. Long ago I wrote a tool to dump Java serialization data (purely structural—without doing any actual classloading), which could be revived and brought up to Java 8, but implementing the complete Object Serialization Stream Protocol is quite tedious. (Also to use such a tool you would need to retain a copy of the InputStream being passed to ObjectInputStream for purposes of retroactive diagnosis, which adds some overhead.) use Reflection for such operation I spent some time playing with MultiClassLoaderSerializer.Input but found no way to obtain better diagnostics, even with reflection. There is simply no useful context available outside stack-allocated variables when resolveClass is called, and no other overridable methods which would yield anything of interest. Print information about rejected classes into a separate file Seems pointless. We already log all rejected classes. That is the most easily available information already. if workdir is enabled Remoting workdir is inapplicable here, since we are talking about the master.
            Hide
            jglick Jesse Glick added a comment -

            I was reluctant to considered this assigned/in review given that my PR does not purport to implement the real improvements listed in the description, but OK.

            Show
            jglick Jesse Glick added a comment - I was reluctant to considered this assigned/in review given that my PR does not purport to implement the real improvements listed in the description, but OK.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/java/hudson/remoting/Channel.java
            src/main/java/hudson/remoting/ClassFilter.java
            src/test/java/hudson/remoting/ChannelFilterTest.java
            http://jenkins-ci.org/commit/remoting/9e6472fbffaf7fc84dd84395725f2db404260def
            Log:
            JENKINS-49027 Better report JEP-200 violations in Remoting (#247)

            • Updated test to look for nested causes.
            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/hudson/remoting/Channel.java src/main/java/hudson/remoting/ClassFilter.java src/test/java/hudson/remoting/ChannelFilterTest.java http://jenkins-ci.org/commit/remoting/9e6472fbffaf7fc84dd84395725f2db404260def Log: JENKINS-49027 Better report JEP-200 violations in Remoting (#247) JENKINS-49027 Better report JEP-200 violations in Remoting. Updated test to look for nested causes.
            Hide
            oleg_nenashev Oleg Nenashev added a comment - - edited

            It has been released in Remoting 3.17 and Jenkins 2.106

            Show
            oleg_nenashev Oleg Nenashev added a comment - - edited It has been released in Remoting 3.17 and Jenkins 2.106

              People

              • Assignee:
                Unassigned
                Reporter:
                oleg_nenashev Oleg Nenashev
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: