Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-7446

CLONE -Gerrit stops sending verifications with the Hudson 1.375 release (1.373 works)

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: git-plugin
    • Labels:
      None
    • Environment:
      CentOS
    • Similar Issues:

      Description

      Hello,

      An RPM update of Hudson yesterday (to 1.375) prevents the gerrit trigger plugin from publishing the results of a build back to Gerrit. We downgraded hudson back to 1.373, and the problem went away. When the build starts, the "Starting build" message does make it to Gerrit, but though the build completes, Gerrit is not notified of the completion, and manual verification of the patch is required.

      Attached is what looks like the relevant hudson log portion. Please let me know if you need anything else.

      Thanks for a great plugin!

      Cheers,

      Bill Shupp

        Attachments

          Issue Links

            Activity

            rsandell rsandell created issue -
            Hide
            rsandell rsandell added a comment -

            It looks like something in the git plugin fails to serialize.
            We can limit the amount of needed save requests in the gerrit-trigger-plugin and catch the exception so we don't break execution, but the root cause still needs to be fixed.

            Show
            rsandell rsandell added a comment - It looks like something in the git plugin fails to serialize. We can limit the amount of needed save requests in the gerrit-trigger-plugin and catch the exception so we don't break execution, but the root cause still needs to be fixed.
            rsandell rsandell made changes -
            Field Original Value New Value
            Link This issue is related to JENKINS-7435 [ JENKINS-7435 ]
            rsandell rsandell made changes -
            Component/s git [ 15543 ]
            Component/s gerrit-trigger [ 15731 ]
            Hide
            abayer Andrew Bayer added a comment -

            Hmm. That's odd - let me try to reproduce this.

            Show
            abayer Andrew Bayer added a comment - Hmm. That's odd - let me try to reproduce this.
            Hide
            abayer Andrew Bayer added a comment -

            So far, no luck reproducing this, but my testbed was deeply messed up, so I've yet to get a useful upgrade case working. Hopefully I'll get there later today.

            Show
            abayer Andrew Bayer added a comment - So far, no luck reproducing this, but my testbed was deeply messed up, so I've yet to get a useful upgrade case working. Hopefully I'll get there later today.
            Hide
            abayer Andrew Bayer added a comment -

            Yeah, definitely can't reproduce this in a non-upgrade case. I'll keep trying.

            Show
            abayer Andrew Bayer added a comment - Yeah, definitely can't reproduce this in a non-upgrade case. I'll keep trying.
            Hide
            abayer Andrew Bayer added a comment -

            Ok, even the upgrade case won't reproduce it. I think something may have been messed up in the reporter's 1.375 install.

            Show
            abayer Andrew Bayer added a comment - Ok, even the upgrade case won't reproduce it. I think something may have been messed up in the reporter's 1.375 install.
            Hide
            rsandell rsandell added a comment -

            Hmm,
            I haven't tried to reproduce it myself yet, just analyzed the log.
            But I am guessing that the scenario is something like:
            1. Two or more projects are triggering on the same change.
            2. One of the builds completes, naturally, a bit before the other(s)
            2.1 When one of the other build completes, data in the first build needs to be changed so ToGerritRunListener et.al. updates the dependency info and does a build.save() on the first build and that's when the NPE gets thrown.

            Show
            rsandell rsandell added a comment - Hmm, I haven't tried to reproduce it myself yet, just analyzed the log. But I am guessing that the scenario is something like: 1. Two or more projects are triggering on the same change. 2. One of the builds completes, naturally, a bit before the other(s) 2.1 When one of the other build completes, data in the first build needs to be changed so ToGerritRunListener et.al. updates the dependency info and does a build.save() on the first build and that's when the NPE gets thrown.
            Hide
            abayer Andrew Bayer added a comment -

            Hmm again - couldn't get that to occur with the scenario you described, but I may well be missing something...

            Show
            abayer Andrew Bayer added a comment - Hmm again - couldn't get that to occur with the scenario you described, but I may well be missing something...
            Hide
            glundh glundh added a comment -

            We can reproduce this error in 1.375 by letting a Gerrit change trigger two git-projects.

            Here is another project that suddenly started to get serialization issues after 1.374:

            http://issues.jenkins-ci.org/browse/JENKINS-7330 (Not sure if it is connected though.)

            When it has happen once during a build, I can reproduce the error by just calling .save() for the affected build in the groovy console:

            println(Hudson.getInstance().getItem("Tools_Git_Clone").getBuildByNumber(19).save());

            java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class hudson.model.FreeStyleBuild
            at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:167)
            at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:135)
            at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:130)
            at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:120)
            at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:94)
            at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
            at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:78)
            at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:63)
            at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:98)
            at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:38)
            at com.thoughtworks.xstream.XStream.marshal(XStream.java:840)
            at com.thoughtworks.xstream.XStream.marshal(XStream.java:829)
            at com.thoughtworks.xstream.XStream.toXML(XStream.java:804)
            at hudson.XmlFile.write(XmlFile.java:165)
            at hudson.model.Run.save(Run.java:1410)
            at sun.reflect.GeneratedMethodAccessor873.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:597)
            at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:229)
            at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:52)
            at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:43)
            at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
            at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120)
            at Script1.run(Script1.groovy:4)
            [CUT]
            Caused by: java.lang.RuntimeException: Failed to serialize hudson.plugins.git.util.BuildData#buildsByBranchName for class hudson.plugins.git.util.BuildData
            at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:167)
            at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:135)
            at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:130)
            at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:120)
            at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:94)
            at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
            at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:78)
            at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:63)
            at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:64)
            at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:55)
            at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
            at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:78)
            at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:63)
            at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:175)
            at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:163)
            ... 85 more
            Caused by: java.lang.NullPointerException
            at java.lang.Class.isAssignableFrom(Native Method)
            at hudson.util.XStream2$1.serializedClass(XStream2.java:115)
            at com.thoughtworks.xstream.mapper.MapperWrapper.serializedClass(MapperWrapper.java:34)
            at com.thoughtworks.xstream.mapper.MapperWrapper.serializedClass(MapperWrapper.java:34)
            at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:58)
            at com.thoughtworks.xstream.converters.collections.MapConverter.marshal(MapConverter.java:57)
            at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
            at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:78)
            at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:63)
            at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:175)
            at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:163)
            ... 99 more

            Show
            glundh glundh added a comment - We can reproduce this error in 1.375 by letting a Gerrit change trigger two git-projects. Here is another project that suddenly started to get serialization issues after 1.374: http://issues.jenkins-ci.org/browse/JENKINS-7330 (Not sure if it is connected though.) When it has happen once during a build, I can reproduce the error by just calling .save() for the affected build in the groovy console: println(Hudson.getInstance().getItem("Tools_Git_Clone").getBuildByNumber(19).save()); java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class hudson.model.FreeStyleBuild at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:167) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:135) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:130) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:120) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:94) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:78) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:63) at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:98) at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:38) at com.thoughtworks.xstream.XStream.marshal(XStream.java:840) at com.thoughtworks.xstream.XStream.marshal(XStream.java:829) at com.thoughtworks.xstream.XStream.toXML(XStream.java:804) at hudson.XmlFile.write(XmlFile.java:165) at hudson.model.Run.save(Run.java:1410) at sun.reflect.GeneratedMethodAccessor873.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:229) at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:52) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:43) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120) at Script1.run(Script1.groovy:4) [CUT] Caused by: java.lang.RuntimeException: Failed to serialize hudson.plugins.git.util.BuildData#buildsByBranchName for class hudson.plugins.git.util.BuildData at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:167) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:135) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:130) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:120) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:94) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:78) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:63) at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:64) at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:55) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:78) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:63) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:175) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:163) ... 85 more Caused by: java.lang.NullPointerException at java.lang.Class.isAssignableFrom(Native Method) at hudson.util.XStream2$1.serializedClass(XStream2.java:115) at com.thoughtworks.xstream.mapper.MapperWrapper.serializedClass(MapperWrapper.java:34) at com.thoughtworks.xstream.mapper.MapperWrapper.serializedClass(MapperWrapper.java:34) at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:58) at com.thoughtworks.xstream.converters.collections.MapConverter.marshal(MapConverter.java:57) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:78) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:63) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:175) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:163) ... 99 more
            Hide
            glundh glundh added a comment -

            It seems this error is due to the buildsByBranchName Hashmap in the hudson.plugins.git.util.BuildData action gets in a state where a key is null:

            println(Hudson.getInstance().getItem("EXPERIMENTAL_Gerrit_Trigger_2").getBuildByNumber(13).getAction(hudson.plugins.git.util.BuildData).buildsByBranchName);

            null:Build #13 of Revision 0f566fdae8a0a3fb06780a976ac41bc97b138383 (null)

            Hudson.getInstance().getItem("EXPERIMENTAL_Gerrit_Trigger_2").getBuildByNumber(13).save();

            java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class hudson.model.FreeStyleBuild
            at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:167)
            at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:135)

            If I clear buildsByBranchName, everything is OK:

            Hudson.getInstance().getItem("EXPERIMENTAL_Gerrit_Trigger_2").getBuildByNumber(13).getAction(hudson.plugins.git.util.BuildData).buildsByBranchName.clear();
            Hudson.getInstance().getItem("EXPERIMENTAL_Gerrit_Trigger_2").getBuildByNumber(13).save();

            <SUCCESS>

            The error in http://issues.jenkins-ci.org/browse/JENKINS-7330 seems also to depend on changes in the serialization logic in 1.374, making Hudson suddenly not accepting a null-string. So I guess they are indeed connected. Maybe we can achieve a similar patch for the GIT-plugin?

            Show
            glundh glundh added a comment - It seems this error is due to the buildsByBranchName Hashmap in the hudson.plugins.git.util.BuildData action gets in a state where a key is null: println(Hudson.getInstance().getItem("EXPERIMENTAL_Gerrit_Trigger_2").getBuildByNumber(13).getAction(hudson.plugins.git.util.BuildData).buildsByBranchName); null:Build #13 of Revision 0f566fdae8a0a3fb06780a976ac41bc97b138383 (null) Hudson.getInstance().getItem("EXPERIMENTAL_Gerrit_Trigger_2").getBuildByNumber(13).save(); java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class hudson.model.FreeStyleBuild at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:167) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:135) If I clear buildsByBranchName, everything is OK: Hudson.getInstance().getItem("EXPERIMENTAL_Gerrit_Trigger_2").getBuildByNumber(13).getAction(hudson.plugins.git.util.BuildData).buildsByBranchName.clear(); Hudson.getInstance().getItem("EXPERIMENTAL_Gerrit_Trigger_2").getBuildByNumber(13).save(); <SUCCESS> The error in http://issues.jenkins-ci.org/browse/JENKINS-7330 seems also to depend on changes in the serialization logic in 1.374, making Hudson suddenly not accepting a null-string. So I guess they are indeed connected. Maybe we can achieve a similar patch for the GIT-plugin?
            Hide
            abayer Andrew Bayer added a comment -

            Ok - looks like changing from HashMap to TreeMap would fix that, but it'll need to translate past builds at upgrade time. Let me write up a patch and send you a link to the hpi to test.

            Show
            abayer Andrew Bayer added a comment - Ok - looks like changing from HashMap to TreeMap would fix that, but it'll need to translate past builds at upgrade time. Let me write up a patch and send you a link to the hpi to test.
            Hide
            dogfood dogfood added a comment -

            Integrated in plugins_hudson-git-plugin #24
            JENKINS-7446 First attempt at getting rid of null keys in buildsByBranchName.

            Andrew Bayer :
            Files :

            • src/main/java/hudson/plugins/git/util/BuildData.java
            Show
            dogfood dogfood added a comment - Integrated in plugins_hudson-git-plugin #24 JENKINS-7446 First attempt at getting rid of null keys in buildsByBranchName. Andrew Bayer : Files : src/main/java/hudson/plugins/git/util/BuildData.java
            Hide
            abayer Andrew Bayer added a comment -

            Actually, even simpler - just a readResolve to switch null keys to "" keys and then some checks to make sure we didn't add new null keys. You can grab the hpi at http://ci.hudson-labs.org/job/plugins_hudson-git-plugin/24/org.jvnet.hudson.plugins$git/

            Show
            abayer Andrew Bayer added a comment - Actually, even simpler - just a readResolve to switch null keys to "" keys and then some checks to make sure we didn't add new null keys. You can grab the hpi at http://ci.hudson-labs.org/job/plugins_hudson-git-plugin/24/org.jvnet.hudson.plugins$git/
            Hide
            rsandell rsandell added a comment -

            Great!
            Will do some tests first thing tomorrow morning.

            Show
            rsandell rsandell added a comment - Great! Will do some tests first thing tomorrow morning.
            Hide
            glundh glundh added a comment -

            The patch work great! Thanks, abayer! However, the GIT build action shows an empty bullet point. Is this supposed to happen? (Not sure if it is related to the patch)

            Screen attached.

            Show
            glundh glundh added a comment - The patch work great! Thanks, abayer! However, the GIT build action shows an empty bullet point. Is this supposed to happen? (Not sure if it is related to the patch) Screen attached.
            Hide
            glundh glundh added a comment - - edited

            File attached.

            Show
            glundh glundh added a comment - - edited File attached.
            glundh glundh made changes -
            Attachment Capture.PNG [ 19757 ]
            Hide
            abayer Andrew Bayer added a comment -

            Lemme see if I can figure that out - I'm guessing that's a side effect of the "" key rather than null key.

            Show
            abayer Andrew Bayer added a comment - Lemme see if I can figure that out - I'm guessing that's a side effect of the "" key rather than null key.
            Hide
            abayer Andrew Bayer added a comment -

            Got it! It was exactly that - I've fixed it now for both BuildData displays - check for the next build after #24 at http://ci.hudson-labs.org/job/plugins_hudson-git-plugin/lastSuccessfulBuild/

            Show
            abayer Andrew Bayer added a comment - Got it! It was exactly that - I've fixed it now for both BuildData displays - check for the next build after #24 at http://ci.hudson-labs.org/job/plugins_hudson-git-plugin/lastSuccessfulBuild/
            Hide
            abayer Andrew Bayer added a comment -

            Resolved in 1.1.

            Show
            abayer Andrew Bayer added a comment - Resolved in 1.1.
            abayer Andrew Bayer made changes -
            Status Open [ 1 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            markewaite Mark Waite made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 137545 ] JNJira + In-Review [ 204528 ]

              People

              • Assignee:
                rsandell rsandell
                Reporter:
                rsandell rsandell
              • Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: