Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-16941

Git publisher post build action fails on slave agent - unable to serialize

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: git-plugin
    • Labels:
      None
    • Environment:
      Linux master server, separate Linux or Windows slave
    • Similar Issues:

      Description

      Using the git plugin to publish a merged change to a central repository previously was successful from either the master node or a slave agent. With the 1.2.0 (and 1.3.0) version of the git plugin it is no longer able to push changes to the central repository after they have merged and are successfully built on the slave. The merge still works as expected when performed on the master node.

      The stack trace reports:

      Commencing build of Revision 363396d76a09a12a2f4b5d94fb4e9981e05ad4a9 (origin/proposals)
      hudson.util.IOException2: remote file operation failed: /var/lib/jenkins/mwaite6-slave/workspace/merge-proposals at hudson.remoting.Channel@4a6ecaf9:fc-agile-2011
      at hudson.FilePath.act(FilePath.java:848)
      at hudson.FilePath.act(FilePath.java:834)
      at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1133)
      at hudson.model.AbstractProject.checkout(AbstractProject.java:1256)
      at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:590)
      at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
      at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:495)
      at hudson.model.Run.execute(Run.java:1502)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:237)
      Caused by: java.io.IOException: Unable to serialize hudson.FilePath$FileCallableWrapper@7c0a37f8
      at hudson.remoting.UserRequest.serialize(UserRequest.java:166)
      at hudson.remoting.UserRequest.<init>(UserRequest.java:62)
      at hudson.remoting.Channel.call(Channel.java:671)
      at hudson.FilePath.act(FilePath.java:841)
      ... 10 more
      Caused by: java.io.NotSerializableException: hudson.model.FreeStyleBuild
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1180)
      at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1528)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1493)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
      at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1528)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1493)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
      at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:346)
      at hudson.remoting.UserRequest._serialize(UserRequest.java:155)
      at hudson.remoting.UserRequest.serialize(UserRequest.java:164)
      ... 13 more

      Steps to recreate the problem:

      • configure a Git repository into which the Jenkins user can push changes
      • create a "master" branch and a "proposals" branch on the Git repository
      • define a Jenkins job which clones the Git repository with ssh protocol
      • configure the Jenkins job git plugin to merge from "/proposals" and "/master" to "master-proposals"
      • restrict the job to execute on any node except master node ("!master")
      • define a post build action to push the resulting merge to the origin/master branch
      • Submit a change to the central git repository on the "proposals" branch
      • Run the Jenkins job to confirm it will combine the master and proposals branches
      • Confirm the Jenkins job fails to push the combined change
        = Serialization exception between master and slave.

      Refer to merge-proposals.xml for the job definition I used in my tests.

        Attachments

          Issue Links

            Activity

            Hide
            sethcall Seth Call added a comment -

            After updating to 1.2.0, we are seeing the exact same error under an identical scenario.

            Downgrading resolved the issue.

            Show
            sethcall Seth Call added a comment - After updating to 1.2.0, we are seeing the exact same error under an identical scenario. Downgrading resolved the issue.
            Hide
            liya Liya Katz added a comment -

            same problem with 1.3.0:
            build has failed on rebase
            Downgrade to 1.1.26 has solved the problem

            Show
            liya Liya Katz added a comment - same problem with 1.3.0: build has failed on rebase Downgrade to 1.1.26 has solved the problem
            Hide
            dwalend David Walend added a comment -

            I see a similar problem with 1.3.0. I get a similar stack trace just by attempting to use "Merge before build" with a branch to merge to. (No post-build steps.)

            Show
            dwalend David Walend added a comment - I see a similar problem with 1.3.0. I get a similar stack trace just by attempting to use "Merge before build" with a branch to merge to. (No post-build steps.)
            Hide
            markewaite Mark Waite added a comment - - edited

            The git-client-plugin 1.0.5 and git-plugin 1.3.0 still show the same problem when attempting a build from a slave agent. The same build works fine if it is restricted to the master node.

            That seems to indicate that this issue is not directly related to whether the git implementation is JGit or command line Git. The git-client-plugin 1.0.4 implementation default was JGit (which showed the problem). The git-client-plugin 1.0.5 implementation default is command line Git (which also shows the problem).

            Show
            markewaite Mark Waite added a comment - - edited The git-client-plugin 1.0.5 and git-plugin 1.3.0 still show the same problem when attempting a build from a slave agent. The same build works fine if it is restricted to the master node. That seems to indicate that this issue is not directly related to whether the git implementation is JGit or command line Git. The git-client-plugin 1.0.4 implementation default was JGit (which showed the problem). The git-client-plugin 1.0.5 implementation default is command line Git (which also shows the problem).
            Hide
            casualt A K added a comment - - edited

            Some testing with my own builds shows that the trigger/difference for causing the "java.io.NotSerializableException" is when the advanced setting "Merge before build" is checked off and filled in. Git operations without this seem to work fine. *Haven't tested extensively of course

            *Edit: This confirms what David Walend commented on earlier.

            Show
            casualt A K added a comment - - edited Some testing with my own builds shows that the trigger/difference for causing the "java.io.NotSerializableException" is when the advanced setting "Merge before build" is checked off and filled in. Git operations without this seem to work fine. *Haven't tested extensively of course *Edit: This confirms what David Walend commented on earlier.
            Hide
            casualt A K added a comment -

            Additionally, I was able to reproduce this behaviour on OSX and Linux slaves, both 64-bit.

            Show
            casualt A K added a comment - Additionally, I was able to reproduce this behaviour on OSX and Linux slaves, both 64-bit.
            Hide
            wooh Adam PAPAI added a comment -

            I've started to find out where went it bad.

            The affected commits are:

            https://github.com/jenkinsci/git-plugin/compare/1349eaba9c5767e9bf52ab27b6c31681f0c13733...6336dc5595266b83c199710953c61351297b7bd8

            so this was the last stable commit:

            https://github.com/jenkinsci/git-plugin/commit/1349eaba9c5767e9bf52ab27b6c31681f0c13733

            The next few commits failed to compile.

            This one compiles, but throws "kind of error", which means it fails to merge:

            Could not checkout release with start point xxxx

            https://github.com/jenkinsci/git-plugin/commit/a60b03fab35e62547a058cc8b4eefd62ee6b05b6

            So somewhere between:

            https://github.com/jenkinsci/git-plugin/compare/1349eaba9c5767e9bf52ab27b6c31681f0c13733...6336dc5595266b83c199710953c61351297b7bd8

            Show
            wooh Adam PAPAI added a comment - I've started to find out where went it bad. The affected commits are: https://github.com/jenkinsci/git-plugin/compare/1349eaba9c5767e9bf52ab27b6c31681f0c13733...6336dc5595266b83c199710953c61351297b7bd8 so this was the last stable commit: https://github.com/jenkinsci/git-plugin/commit/1349eaba9c5767e9bf52ab27b6c31681f0c13733 The next few commits failed to compile. This one compiles, but throws "kind of error", which means it fails to merge: Could not checkout release with start point xxxx https://github.com/jenkinsci/git-plugin/commit/a60b03fab35e62547a058cc8b4eefd62ee6b05b6 So somewhere between: https://github.com/jenkinsci/git-plugin/compare/1349eaba9c5767e9bf52ab27b6c31681f0c13733...6336dc5595266b83c199710953c61351297b7bd8
            Hide
            markewaite Mark Waite added a comment -

            Thanks for spending the effort to seek the failing submission. Unfortunately, I think that the actual problem change is hidden in the refactoring which abstracted the Git implementation into an interface, then provided two alternate implementations of that interface, one using the command line Git, the other using JGit. Initially, the default implementation was JGit. With git-client-plugin 1.0.5, the default implementation is now command line Git.

            I think the problem was introduced by the refactoring because the problem still exists even after the default implementation was switched from JGit to command line git.

            The high level history (as far as I can see), looks something like this:

            • Git plugin 1.1.26 - last version known able to push a merged submission back from a slave agent
            • Git plugin 1.2.0 + Git client plugin 1.0.1 through 1.0.4 - refactored to provide a Git interface, default implementation with JGit
            • Git plugin 1.3.0 + Git client plugin 1.0.5 - bug fixes and switch default implementation to command line Git

            Part of me makes the wild guess that this issue would be resolved if we could understand why it cannot serialize. I don't understand the architecture of the slave to master communication, but I thought it was defined by the Java classes and was implemented with Java serialization. My wild guess was that some field or fields were "lost" or "redefined" in the refactoring which break the serialization between the master and slave. However, that is truly just wild guessing on my part.

            Show
            markewaite Mark Waite added a comment - Thanks for spending the effort to seek the failing submission. Unfortunately, I think that the actual problem change is hidden in the refactoring which abstracted the Git implementation into an interface, then provided two alternate implementations of that interface, one using the command line Git, the other using JGit. Initially, the default implementation was JGit. With git-client-plugin 1.0.5, the default implementation is now command line Git. I think the problem was introduced by the refactoring because the problem still exists even after the default implementation was switched from JGit to command line git. The high level history (as far as I can see), looks something like this: Git plugin 1.1.26 - last version known able to push a merged submission back from a slave agent Git plugin 1.2.0 + Git client plugin 1.0.1 through 1.0.4 - refactored to provide a Git interface, default implementation with JGit Git plugin 1.3.0 + Git client plugin 1.0.5 - bug fixes and switch default implementation to command line Git Part of me makes the wild guess that this issue would be resolved if we could understand why it cannot serialize. I don't understand the architecture of the slave to master communication, but I thought it was defined by the Java classes and was implemented with Java serialization. My wild guess was that some field or fields were "lost" or "redefined" in the refactoring which break the serialization between the master and slave. However, that is truly just wild guessing on my part.
            Hide
            wooh Adam PAPAI added a comment -

            Meanwhile I think I've found the problematic part. It now works for me locally. Trying to create a patch.

            Show
            wooh Adam PAPAI added a comment - Meanwhile I think I've found the problematic part. It now works for me locally. Trying to create a patch.
            Hide
            markewaite Mark Waite added a comment -

            That sounds great! If you'd like additional testing of your patch, I'd be happy to help over the weekend. If you submit it as a pull request to the plugin repository, I can pull it into my repository as well and build it locally for my testing.

            Show
            markewaite Mark Waite added a comment - That sounds great! If you'd like additional testing of your patch, I'd be happy to help over the weekend. If you submit it as a pull request to the plugin repository, I can pull it into my repository as well and build it locally for my testing.
            Hide
            wooh Adam PAPAI added a comment - - edited

            This one breaks the feature. At last for us it breaks when it wants to merge the main with another branch. (so merge before build)

            https://github.com/cosmin/git-plugin/commit/a7a9e3d714adc80eab8a9271c79883b6cdf7bd93

            Try to revert it and test it on your system.

            Let me know how it works.

            Show
            wooh Adam PAPAI added a comment - - edited This one breaks the feature. At last for us it breaks when it wants to merge the main with another branch. (so merge before build) https://github.com/cosmin/git-plugin/commit/a7a9e3d714adc80eab8a9271c79883b6cdf7bd93 Try to revert it and test it on your system. Let me know how it works.
            Hide
            wooh Adam PAPAI added a comment -

            I've tested it.

            It works now. It can check out on a remote slave, it can merge to another repo, and after the tests it can push the merged thing to another branch.

            So the commit above my previous commit did it. I reverted it. Created a pull request for that.

            Show
            wooh Adam PAPAI added a comment - I've tested it. It works now. It can check out on a remote slave, it can merge to another repo, and after the tests it can push the merged thing to another branch. So the commit above my previous commit did it. I reverted it. Created a pull request for that.
            Hide
            jglick Jesse Glick added a comment -

            Since https://github.com/jenkinsci/git-plugin/pull/148 has been merged, should this not be set to Resolved? (I know no plugin release containing the fix has yet been made, but it is conventional to mark bugs closed once the fix has been committed to the master branch.)

            Show
            jglick Jesse Glick added a comment - Since https://github.com/jenkinsci/git-plugin/pull/148 has been merged, should this not be set to Resolved? (I know no plugin release containing the fix has yet been made, but it is conventional to mark bugs closed once the fix has been committed to the master branch.)
            Hide
            markewaite Mark Waite added a comment -

            Agreed with @jglick that since the change has been submitted, this should be resolved.

            Show
            markewaite Mark Waite added a comment - Agreed with @jglick that since the change has been submitted, this should be resolved.
            Hide
            markewaite Mark Waite added a comment - - edited

            Nicolas just pushed a release of git-client-plugin 1.0.6 and git-plugin 1.4.0. The git-client-plugin release is already available on the updates site. The git-plugin release should be available soon.

            Show
            markewaite Mark Waite added a comment - - edited Nicolas just pushed a release of git-client-plugin 1.0.6 and git-plugin 1.4.0. The git-client-plugin release is already available on the updates site. The git-plugin release should be available soon.
            Hide
            markewaite Mark Waite added a comment -

            Confirmed fixed with git-client-plugin 1.0.6 and git-plugin 1.4.0.

            Show
            markewaite Mark Waite added a comment - Confirmed fixed with git-client-plugin 1.0.6 and git-plugin 1.4.0.

              People

              • Assignee:
                ndeloof Nicolas De Loof
                Reporter:
                markewaite Mark Waite
              • Votes:
                7 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: