Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-6144

svn log copier thread is never destroyed if an SVNException is thrown.

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: subversion-plugin
    • Labels:
      None
    • Environment:
      Windows Server 2003 R2 32 bit, JDK 1.6.0_18, Hudson 1.352, Subversion plugin 1.16
    • Similar Issues:

      Description

      unfortunately in the past weeks the SVN service we are using has become a bit unreliable. this resulted in about 5-10 errors per hours (socket read timeout, etc) mostly during SCM poll triggered builds.

      The outcome of this error leaves 'svn log copier' thread hanging around in a TIMED_WAITING state, These threads keep growing in number until an inevitable JVM crash with an OOME due to resource starvation (GrowableArray), jdk 1.6.0 update is out and will probably delay further this crash but I think it's better to fix the issue in the first place.

      I can provide further details and if if required.

      thanks

        Attachments

          Activity

          Hide
          abarbieri Andrea Barbieri added a comment -

          I can provide further details and info if required

          Show
          abarbieri Andrea Barbieri added a comment - I can provide further details and info if required
          Hide
          abarbieri Andrea Barbieri added a comment -

          with a simple proof of concept using the subversion plugin 1.16, if I:

          • create a simple free style project
          • check out an (existing) SVN module
          • trigger a build every 2 minutes

          there are no 'svn log copier' threads reported by the javamelody monitoring plugin.

          If now I simply reconfigure the job by mistyping the SVN URL I will get a failed build every time the job get triggered with a console output log looking like this:
          Started by timer
          Location 'https://svn.example.com/repos/enforcer/test3/trunk' does not exist
          Checking out https://svn.example.com/repos/enforcer/test3/trunk
          ERROR: Failed to check out https://svn.example.com/repos/enforcer/test3/trunk
          org.tmatesoft.svn.core.SVNException: svn: URL 'https://svn.example.com/repos/enforcer/test3/trunk' doesn't exist
          at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64)
          at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51)
          at org.tmatesoft.svn.core.wc.SVNUpdateClient.doCheckout(SVNUpdateClient.java:902)
          at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:740)
          at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:660)
          at hudson.FilePath.act(FilePath.java:747)
          at hudson.FilePath.act(FilePath.java:729)
          at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:653)
          at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:601)
          at hudson.model.AbstractProject.checkout(AbstractProject.java:1040)
          at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
          at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
          at hudson.model.Run.run(Run.java:1243)
          at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
          at hudson.model.ResourceController.execute(ResourceController.java:88)
          at hudson.model.Executor.run(Executor.java:122)
          Finished: FAILURE

          and most importantly every time a failed build happens the number of existing 'svn log copier' threads is incremented by one.

          If I apply the attached patch (not the most elegant one!, just to prove the point) to the subversion plugin (SubversionSCM.java) and update the plugin used by the hudson instance, even when the builds fail no 'svn log copier' thread persists in a TIMED_WAITING state (and the total number of threads never grows indefinitely - or as it happens never crashes the JVM instance due to resources exhaustion).

          Show
          abarbieri Andrea Barbieri added a comment - with a simple proof of concept using the subversion plugin 1.16, if I: create a simple free style project check out an (existing) SVN module trigger a build every 2 minutes there are no 'svn log copier' threads reported by the javamelody monitoring plugin. If now I simply reconfigure the job by mistyping the SVN URL I will get a failed build every time the job get triggered with a console output log looking like this: Started by timer Location 'https://svn.example.com/repos/enforcer/test3/trunk' does not exist Checking out https://svn.example.com/repos/enforcer/test3/trunk ERROR: Failed to check out https://svn.example.com/repos/enforcer/test3/trunk org.tmatesoft.svn.core.SVNException: svn: URL 'https://svn.example.com/repos/enforcer/test3/trunk' doesn't exist at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64) at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51) at org.tmatesoft.svn.core.wc.SVNUpdateClient.doCheckout(SVNUpdateClient.java:902) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:740) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:660) at hudson.FilePath.act(FilePath.java:747) at hudson.FilePath.act(FilePath.java:729) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:653) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:601) at hudson.model.AbstractProject.checkout(AbstractProject.java:1040) at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411) at hudson.model.Run.run(Run.java:1243) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:122) Finished: FAILURE and most importantly every time a failed build happens the number of existing 'svn log copier' threads is incremented by one. If I apply the attached patch (not the most elegant one!, just to prove the point) to the subversion plugin (SubversionSCM.java) and update the plugin used by the hudson instance, even when the builds fail no 'svn log copier' thread persists in a TIMED_WAITING state (and the total number of threads never grows indefinitely - or as it happens never crashes the JVM instance due to resources exhaustion).
          Hide
          abarbieri Andrea Barbieri added a comment -

          StreamCopyThread runaway thread patch.

          Show
          abarbieri Andrea Barbieri added a comment - StreamCopyThread runaway thread patch.
          Hide
          abarbieri Andrea Barbieri added a comment -

          some feedback after a week with the patched subversion plugin. No more JVM OOME crashes due to orphaned 'svn log copier' threads creep.

          Show
          abarbieri Andrea Barbieri added a comment - some feedback after a week with the patched subversion plugin. No more JVM OOME crashes due to orphaned 'svn log copier' threads creep.
          Show
          sogabe sogabe added a comment - Fixed in 1.7. See http://fisheye.jenkins-ci.org/changelog/Hudson/trunk/hudson/plugins/subversion?cs=30194
          Hide
          abarbieri Andrea Barbieri added a comment -

          1.17

          any idea when it will be formally released?

          and... thanks a lot!

          Show
          abarbieri Andrea Barbieri added a comment - 1.17 any idea when it will be formally released? and... thanks a lot!
          Hide
          abarbieri Andrea Barbieri added a comment -

          when building the latest 1.17 snapshot version of the plugin I get the following error:
          mvn clean package -Dmaven.test.skip
          [INFO] Scanning for projects...
          [INFO] ------------------------------------------------------------------------
          [INFO] Building Hudson Subversion Plug-in
          [INFO] task-segment: [clean, package]
          [INFO] ------------------------------------------------------------------------
          [INFO] [clean:clean

          {execution: default-clean}

          ]
          [INFO] [enforcer:display-info

          {execution: default}

          ]
          [INFO] Maven Version: 2.2.1
          [INFO] JDK Version: 1.6.0_15 normalized as: 1.6.0-15
          [INFO] OS Info: Arch: amd64 Family: unix Name: linux Version: 2.6.31-20-generic
          [INFO] [localizer:generate

          {execution: default}

          ]
          [INFO] [resources:resources

          {execution: default-resources}

          ]
          [WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
          [INFO] Copying 168 resources
          [INFO] [hpi:apt-compile

          {execution: default-apt-compile}

          ]
          [INFO] Compiling 25 source files to /media/LG_EXT_HDD/sandbox/hudson/hudson/plugins/subversion/target/classes
          Note: Generating hudson/scm/SubversionChangeLogSet/RevisionInfo.javadoc
          Note: Generating hudson/scm/SubversionChangeLogSet.javadoc
          Note: Generating hudson/scm/SubversionSCM.javadoc
          Note: Generating hudson/scm/SubversionChangeLogSet/Path.javadoc
          Note: Generating hudson/scm/SubversionTagAction/TagInfo.javadoc
          Note: Generating hudson/scm/SubversionChangeLogSet/LogEntry.javadoc
          Note: Generating hudson/scm/SubversionTagAction.javadoc
          Note: Generating hudson/scm/SubversionSCM/ModuleLocation.javadoc
          Note: Generating hudson/scm/browsers/FishEyeSVN.stapler
          Note: Generating hudson/scm/browsers/Sventon2.stapler
          Note: Generating hudson/scm/browsers/ViewSVN.stapler
          Note: Generating hudson/scm/browsers/CollabNetSVN.stapler
          Note: Generating hudson/scm/SubversionSCM/ModuleLocation.stapler
          Note: Generating hudson/scm/SubversionSCM.stapler
          Note: Generating hudson/scm/browsers/Sventon.stapler
          Note: Generating hudson/scm/browsers/WebSVN.stapler
          Note: Generating hudson/scm/browsers/FishEyeSVN/DescriptorImpl/doCheckUrl.stapler
          Note: Generating hudson/scm/browsers/Sventon2/DescriptorImpl/doCheckUrl.stapler
          Note: Generating hudson/scm/SubversionSCM/DescriptorImpl/doCheckRemote.stapler
          Note: Generating hudson/scm/SubversionSCM/DescriptorImpl/doCheckLocal.stapler
          Note: Generating hudson/scm/SubversionSCM/DescriptorImpl/doCheckExcludedRegions.stapler
          Note: Generating hudson/scm/SubversionSCM/DescriptorImpl/doCheckIncludedRegions.stapler
          Note: Generating hudson/scm/SubversionSCM/DescriptorImpl/doCheckExcludedUsers.stapler
          Note: Generating hudson/scm/SubversionSCM/DescriptorImpl/doCheckExcludedCommitMessages.stapler
          Note: Generating hudson/scm/SubversionSCM/DescriptorImpl/doCheckRevisionPropertiesSupported.stapler
          Note: Generating hudson/scm/browsers/Sventon/DescriptorImpl/doCheckUrl.stapler
          Note: hudson.scm.browsers.FishEyeSVN.DescriptorImpl indexed under hudson.Extension
          Note: hudson.scm.browsers.Sventon2.DescriptorImpl indexed under hudson.Extension
          Note: hudson.scm.browsers.ViewSVN.DescriptorImpl indexed under hudson.Extension
          Note: hudson.scm.SubversionMailAddressResolverImpl indexed under hudson.Extension
          Note: hudson.scm.browsers.CollabNetSVN.DescriptorImpl indexed under hudson.Extension
          Note: hudson.scm.SubversionStatus indexed under hudson.Extension
          Note: hudson.scm.SubversionSCM.DescriptorImpl indexed under hudson.Extension
          Note: hudson.scm.browsers.Sventon.DescriptorImpl indexed under hudson.Extension
          Note: hudson.scm.browsers.WebSVN.DescriptorImpl indexed under hudson.Extension
          Note: hudson.scm.SubversionTagAction.DescriptorImpl indexed under hudson.Extension
          /path/to/sandbox/hudson/hudson/plugins/subversion/src/main/java/hudson/scm/SubversionSCM.java:465: method does not override or implement a method from a supertype
          @Override
          ^
          Note: Some input files use unchecked or unsafe operations.
          Note: Recompile with -Xlint:unchecked for details.
          1 error
          [INFO] ------------------------------------------------------------------------
          [ERROR] BUILD ERROR
          [INFO] ------------------------------------------------------------------------
          [INFO] Fatal error compiling

          Embedded error: APT failed: 1
          [INFO] ------------------------------------------------------------------------
          [INFO] For more information, run Maven with the -e switch
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 23 seconds
          [INFO] Finished at: Mon Apr 19 17:34:58 BST 2010
          [INFO] Final Memory: 73M/423M
          [INFO] ------------------------------------------------------------------------

          Show
          abarbieri Andrea Barbieri added a comment - when building the latest 1.17 snapshot version of the plugin I get the following error: mvn clean package -Dmaven.test.skip [INFO] Scanning for projects... [INFO] ------------------------------------------------------------------------ [INFO] Building Hudson Subversion Plug-in [INFO] task-segment: [clean, package] [INFO] ------------------------------------------------------------------------ [INFO] [clean:clean {execution: default-clean} ] [INFO] [enforcer:display-info {execution: default} ] [INFO] Maven Version: 2.2.1 [INFO] JDK Version: 1.6.0_15 normalized as: 1.6.0-15 [INFO] OS Info: Arch: amd64 Family: unix Name: linux Version: 2.6.31-20-generic [INFO] [localizer:generate {execution: default} ] [INFO] [resources:resources {execution: default-resources} ] [WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent! [INFO] Copying 168 resources [INFO] [hpi:apt-compile {execution: default-apt-compile} ] [INFO] Compiling 25 source files to /media/LG_EXT_HDD/sandbox/hudson/hudson/plugins/subversion/target/classes Note: Generating hudson/scm/SubversionChangeLogSet/RevisionInfo.javadoc Note: Generating hudson/scm/SubversionChangeLogSet.javadoc Note: Generating hudson/scm/SubversionSCM.javadoc Note: Generating hudson/scm/SubversionChangeLogSet/Path.javadoc Note: Generating hudson/scm/SubversionTagAction/TagInfo.javadoc Note: Generating hudson/scm/SubversionChangeLogSet/LogEntry.javadoc Note: Generating hudson/scm/SubversionTagAction.javadoc Note: Generating hudson/scm/SubversionSCM/ModuleLocation.javadoc Note: Generating hudson/scm/browsers/FishEyeSVN.stapler Note: Generating hudson/scm/browsers/Sventon2.stapler Note: Generating hudson/scm/browsers/ViewSVN.stapler Note: Generating hudson/scm/browsers/CollabNetSVN.stapler Note: Generating hudson/scm/SubversionSCM/ModuleLocation.stapler Note: Generating hudson/scm/SubversionSCM.stapler Note: Generating hudson/scm/browsers/Sventon.stapler Note: Generating hudson/scm/browsers/WebSVN.stapler Note: Generating hudson/scm/browsers/FishEyeSVN/DescriptorImpl/doCheckUrl.stapler Note: Generating hudson/scm/browsers/Sventon2/DescriptorImpl/doCheckUrl.stapler Note: Generating hudson/scm/SubversionSCM/DescriptorImpl/doCheckRemote.stapler Note: Generating hudson/scm/SubversionSCM/DescriptorImpl/doCheckLocal.stapler Note: Generating hudson/scm/SubversionSCM/DescriptorImpl/doCheckExcludedRegions.stapler Note: Generating hudson/scm/SubversionSCM/DescriptorImpl/doCheckIncludedRegions.stapler Note: Generating hudson/scm/SubversionSCM/DescriptorImpl/doCheckExcludedUsers.stapler Note: Generating hudson/scm/SubversionSCM/DescriptorImpl/doCheckExcludedCommitMessages.stapler Note: Generating hudson/scm/SubversionSCM/DescriptorImpl/doCheckRevisionPropertiesSupported.stapler Note: Generating hudson/scm/browsers/Sventon/DescriptorImpl/doCheckUrl.stapler Note: hudson.scm.browsers.FishEyeSVN.DescriptorImpl indexed under hudson.Extension Note: hudson.scm.browsers.Sventon2.DescriptorImpl indexed under hudson.Extension Note: hudson.scm.browsers.ViewSVN.DescriptorImpl indexed under hudson.Extension Note: hudson.scm.SubversionMailAddressResolverImpl indexed under hudson.Extension Note: hudson.scm.browsers.CollabNetSVN.DescriptorImpl indexed under hudson.Extension Note: hudson.scm.SubversionStatus indexed under hudson.Extension Note: hudson.scm.SubversionSCM.DescriptorImpl indexed under hudson.Extension Note: hudson.scm.browsers.Sventon.DescriptorImpl indexed under hudson.Extension Note: hudson.scm.browsers.WebSVN.DescriptorImpl indexed under hudson.Extension Note: hudson.scm.SubversionTagAction.DescriptorImpl indexed under hudson.Extension /path/to/sandbox/hudson/hudson/plugins/subversion/src/main/java/hudson/scm/SubversionSCM.java:465: method does not override or implement a method from a supertype @Override ^ Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. 1 error [INFO] ------------------------------------------------------------------------ [ERROR] BUILD ERROR [INFO] ------------------------------------------------------------------------ [INFO] Fatal error compiling Embedded error: APT failed: 1 [INFO] ------------------------------------------------------------------------ [INFO] For more information, run Maven with the -e switch [INFO] ------------------------------------------------------------------------ [INFO] Total time: 23 seconds [INFO] Finished at: Mon Apr 19 17:34:58 BST 2010 [INFO] Final Memory: 73M/423M [INFO] ------------------------------------------------------------------------
          Hide
          sogabe sogabe added a comment -

          updated version of Hudson to 1.355.

          Show
          sogabe sogabe added a comment - updated version of Hudson to 1.355.

            People

            • Assignee:
              sogabe sogabe
              Reporter:
              abarbieri Andrea Barbieri
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: