Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-8056

Hudson stops handling Subversion post commit hooks properly after a Hudson upgrade (from 1.372 to 1.381)

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Blocker
    • Resolution: Fixed
    • Component/s: core, subversion-plugin
    • Labels:
      None
    • Environment:
      Linux kriek 2.6.26-2-amd64 #1 SMP Sun Jun 20 20:16:30 UTC 2010 x86_64 GNU/Linux
      java version "1.6.0_19"
      Java(TM) SE Runtime Environment (build 1.6.0_19-b04)
      Java HotSpot(TM) 64-Bit Server VM (build 16.2-b04, mixed mode)
    • Similar Issues:

      Description

      A few weeks ago we upgraded from 1.372 to 1.381 which solved a significant amount of issues (many thanks for that!).
      Along with the upgrade of Hudson, we upgraded the Subversion Hudson plugin from 1.18 to 1.20.
      Shortly after the upgrade of these two, we started noticing that Hudson was not triggering builds, despite the fact that Subversion was indeed notifying it of commits. We have the SCM Subversion polling set to:

      */30 * * * *
      

      thus, almost completely relying on the post-commit-hook to trigger the build (as opposed to the SCM polling getting stuffed up gradually).

      The Subversion post-commit-hook works correctly and is sending a request, as can be seen below.
      This is what the successful output should look like:

      --17:30:49--  http://hudson:8082/hudson/subversion/c5784c02-6b0d-0410-910a-a7a69de433da/notifyCommit?rev=71423
                 => `/tmp/svn-notify-commit-post'
      Resolving hudson... 192.168.0.41
      Connecting to hudson[192.168.0.41]:8082... connected.
      HTTP request sent, awaiting response... 200 OK
      Length: 0
      
          0K                                                           0.00 B/s
      

      This is the sort of output we currently have:

      --18:09:09--  http://hudson:8082/hudson/subversion/c5784c02-6b0d-0410-910a-a7a69de433da/notifyCommit?rev=71424
                 => `/tmp/svn-notify-commit-post'
      Resolving hudson... 192.168.0.41
      Connecting to hudson[192.168.0.41]:8082... connected.
      HTTP request sent, awaiting response... 500 READER
      18:09:09 ERROR 500: READER.
      

      This is an excerpt from the Jetty log:

      2010-11-03 14:09:56.888:WARN:/hudson:Error while serving http://hudson:8082/hudson/subversion/c5784c02-6b0d-0410-910a-a7a69de433da/notifyCommit
      java.lang.reflect.InvocationTargetException
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:259)
      	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:126)
      	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:65)
      	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:79)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:30)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:537)
      	at org.kohsuke.stapler.MetaClass$13.dispatch(MetaClass.java:359)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:537)
      	at org.kohsuke.stapler.MetaClass$13.dispatch(MetaClass.java:359)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:537)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:453)
      	at org.kohsuke.stapler.Stapler.service(Stapler.java:135)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:527)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1216)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:94)
      	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:86)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1187)
      	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1187)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1187)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:421)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:456)
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:924)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:358)
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:860)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
      	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:245)
      	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113)
      	at org.eclipse.jetty.server.Server.handle(Server.java:335)
      	at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:588)
      	at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:1046)
      	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:764)
      	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:217)
      	at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:418)
      	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:489)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)
      	at java.lang.Thread.run(Thread.java:619)
      2010-11-03 14:09:56.889:WARN::/hudson/subversion/c5784c02-6b0d-0410-910a-a7a69de433da/notifyCommit
      java.lang.IllegalStateException: READER
      	at org.eclipse.jetty.server.Request.getInputStream(Request.java:508)
      	at org.eclipse.jetty.server.Request.extractParameters(Request.java:251)
      	at org.eclipse.jetty.server.Request.getParameter(Request.java:648)
      	at javax.servlet.ServletRequestWrapper.getParameter(ServletRequestWrapper.java:169)
      	at hudson.scm.SubversionRepositoryStatus.doNotifyCommit(SubversionRepositoryStatus.java:84)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:259)
      	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:126)
      	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:65)
      	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:79)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:30)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:537)
      	at org.kohsuke.stapler.MetaClass$13.dispatch(MetaClass.java:359)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:537)
      	at org.kohsuke.stapler.MetaClass$13.dispatch(MetaClass.java:359)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:537)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:453)
      	at org.kohsuke.stapler.Stapler.service(Stapler.java:135)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:527)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1216)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:94)
      	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:86)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1187)
      	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1187)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1187)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:421)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:456)
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:924)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:358)
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:860)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
      	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:245)
      	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113)
      	at org.eclipse.jetty.server.Server.handle(Server.java:335)
      	at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:588)
      	at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:1046)
      	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:764)
      	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:217)
      	at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:418)
      	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:489)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)
      	at java.lang.Thread.run(Thread.java:619)
      Nov 3, 2010 2:11:03 PM hudson.scm.PerJobCredentialStore <init>
      INFO: Failed to retrieve Subversion credentials from /local/hudson/jobs/FAS-6199/subversion.credentials
      hudson.util.IOException2: Unable to read /local/hudson/jobs/FAS-6199/subversion.credentials
      	at hudson.XmlFile.unmarshal(XmlFile.java:152)
      	at hudson.scm.PerJobCredentialStore.<init>(PerJobCredentialStore.java:38)
      	at hudson.scm.SubversionSCM$DescriptorImpl.createAuthenticationProvider(SubversionSCM.java:1659)
      	at hudson.scm.SubversionSCM$DescriptorImpl.getRepository(SubversionSCM.java:1858)
      	at hudson.scm.SubversionSCM$DescriptorImpl.checkRepositoryPath(SubversionSCM.java:1831)
      	at hudson.scm.SubversionSCM.repositoryLocationsNoLongerExist(SubversionSCM.java:1998)
      	at hudson.scm.SubversionSCM.compareRemoteRevisionWith(SubversionSCM.java:1045)
      	at hudson.scm.SCM._compareRemoteRevisionWith(SCM.java:353)
      	at hudson.scm.SCM.poll(SCM.java:370)
      	at hudson.model.AbstractProject.poll(AbstractProject.java:1218)
      	at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:417)
      	at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:446)
      	at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:619)
      Caused by: com.thoughtworks.xstream.converters.ConversionException: No InvocationHandler specified for dynamic proxy
      ---- Debugging information ----
      class               : com.thoughtworks.xstream.mapper.DynamicProxyMapper$DynamicProxy
      required-type       : com.thoughtworks.xstream.mapper.DynamicProxyMapper$DynamicProxy
      path                : /hudson.scm.PerJobCredentialStore
      line number         : 13
      -------------------------------
      	at com.thoughtworks.xstream.converters.extended.DynamicProxyConverter.unmarshal(DynamicProxyConverter.java:92)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:82)
      	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:63)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:76)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:60)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:137)
      	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:33)
      	at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:926)
      	at hudson.util.XStream2.unmarshal(XStream2.java:80)
      	at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:912)
      	at hudson.XmlFile.unmarshal(XmlFile.java:148)
      	... 18 more
      

      The error above looks like a problem with the deserialization of the per-module subversion.credentials file below:

      <?xml version='1.0' encoding='UTF-8'?>
      <hudson.scm.PerJobCredentialStore resolves-to="dynamic-proxy">
        <h class="hudson.remoting.RemoteInvocationHandler" serialization="custom">
          <hudson.remoting.RemoteInvocationHandler>
            <default>
              <autoUnexportByCaller>false</autoUnexportByCaller>
              <goingHome>false</goingHome>
              <oid>9271</oid>
              <userProxy>true</userProxy>
            </default>
          </hudson.remoting.RemoteInvocationHandler>
        </h>
      </hudson.scm.PerJobCredentialStore>
      

      Rolling back to 1.18 did not solve the problem. In addition, I tried the following:

      • Create a new job which checks out a project from the same Subversion repository.
      • Commit a changed file.
        The build worked and got triggered. So then I figured I should stop Hudson and write a simple script to replace all per-project credential files (they all have the same content, except there is a serious difference between a working one and one of an older project). This did not work. For some reason Hudson would just re-overwrite these files with the old broken ones. Below you can see a subversion.credentials file for a working module:
        <?xml version='1.0' encoding='UTF-8'?>
        <hudson.scm.PerJobCredentialStore>
          <credentials class="hashtable">
            <entry>
              <string>svn+ssh://subversion</string>
              <hudson.scm.SubversionSCM_-DescriptorImpl_-SshPublicKeyCredential>
                <userName>some_user</userName>
                <passphrase></passphrase>
                <id>85c531bf25d7ae61</id>
              </hudson.scm.SubversionSCM_-DescriptorImpl_-SshPublicKeyCredential>
            </entry>
          </credentials>
        </hudson.scm.PerJobCredentialStore>
        

      As you can see the above files have little in common.

      I have also tried upgrading to 1.384. That didn't fix it either and actually caused nodes to die, so I rolled back to 1.381.

      I have also looked at the PerJobCredentialStore class. I didn't find anything suspicious there (in fact, if I recall correctly, this class hasn't been changed in about half a year).

      A fix for this would be highly appreciated, as we have some 200 modules which now rely solely on a build triggered every half hour.

      Please advise and many thanks in advance,

      Martin

        Attachments

          Activity

          Hide
          carlspring carlspring added a comment -

          For those hitting this issue, this is a workaround:

          1. Check out version 1.17 from:
            https://svn.dev.java.net/svn/hudson/tags/subversion-1.7
            
          2. Build it with:
            mvn clean package
            
          3. Shutdown Hudson.
          4. Backup and remove the old plugin ($HUDSON_HOME/plugins/subversion*).
          5. Most-likely optional, but just to make sure there are no broken subversion.credential files lying around:
            rm -rf `find $HUDSON_HOME/jobs -maxdepth 2 -name subversion.credentials -type f`
            
          6. Copy the compiled subversion plugin (version 1.17) to $HUDSON_HOME/plugins.
          7. Start Hudson.
          Show
          carlspring carlspring added a comment - For those hitting this issue, this is a workaround: Check out version 1.17 from: https: //svn.dev.java.net/svn/hudson/tags/subversion-1.7 Build it with: mvn clean package Shutdown Hudson. Backup and remove the old plugin ($HUDSON_HOME/plugins/subversion*). Most-likely optional, but just to make sure there are no broken subversion.credential files lying around: rm -rf `find $HUDSON_HOME/jobs -maxdepth 2 -name subversion.credentials -type f` Copy the compiled subversion plugin (version 1.17) to $HUDSON_HOME/plugins. Start Hudson.
          Hide
          carlspring carlspring added a comment -

          In addition, we only had the option to rollback from 1.20 to 1.18. There was no option to rollback to an earlier version.

          Show
          carlspring carlspring added a comment - In addition, we only had the option to rollback from 1.20 to 1.18. There was no option to rollback to an earlier version.
          Hide
          kohsuke Kohsuke Kawaguchi added a comment -

          This problem appears to be container specific.
          Fixed in rev.37937 toward Subversion plugin 1.23.

          Show
          kohsuke Kohsuke Kawaguchi added a comment - This problem appears to be container specific. Fixed in rev.37937 toward Subversion plugin 1.23.
          Hide
          kohsuke Kohsuke Kawaguchi added a comment -

          This issue is unrelated to JENKINS-8061.

          Show
          kohsuke Kohsuke Kawaguchi added a comment - This issue is unrelated to JENKINS-8061 .

            People

            • Assignee:
              kohsuke Kohsuke Kawaguchi
              Reporter:
              carlspring carlspring
            • Votes:
              2 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: