Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-10222

Jenkins cleaning out workspace unnecessarily when polling SCM (SVN)

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: subversion-plugin
    • Labels:
      None
    • Environment:
      Windows

      Description

      I am using a SVN repository which is being checked out at the beginning of every build in a Jenkins job. It is set to "poll SCM" for updates. However, regardless of whether there were changes or not it is cleaning out the workspace and downloading the whole repository in it's entirety every time which is unnecessary.

      Builds are taking much longer than they should as the SVN poll should only be checking out newly-updated files. Sample log:

      Checking out a fresh workspace because the workspace is not file://PICKLE/Repositories/project
      Cleaning workspace.....

        Issue Links

          Activity

          Hide
          laymanmb Matt Layman added a comment -

          This error message comes from the hudson.scm.subversion.UpdateUpdater.java at line 96. The reason for the failure is because svnInfo.url != url. So I built a test version of the plugin and logged some extra information so I could find out what 'svnInfo.url' was and what 'url' was.

          My job configuration uses the 'svn update' option and my SVN URL looks something like this (I've had to modify the URL because the path is on an intranet where info is proprietary):

          https://<some IP address>/svn/REPO/directory/branches/some_branch/@HEAD

          In this scenario, I got the following:

          • svnInfo.url = https://<some IP address>/svn/REPO/directory/branches/some_branch
          • url = https://<some IP address>/svn/REPO/directory/branches/some_branch/

          The observable difference is that svnInfo.url has no trailing slash and url has a trailing slash.

          Therefore, I presume that the fix is to modify one of those two variables in some appropriate manner so that they agree with each other. If I have time to investigate the appropriate place to add (or remove) the slash, I'll report back.

          Show
          laymanmb Matt Layman added a comment - This error message comes from the hudson.scm.subversion.UpdateUpdater.java at line 96. The reason for the failure is because svnInfo.url != url. So I built a test version of the plugin and logged some extra information so I could find out what 'svnInfo.url' was and what 'url' was. My job configuration uses the 'svn update' option and my SVN URL looks something like this (I've had to modify the URL because the path is on an intranet where info is proprietary): https://<some IP address>/svn/REPO/directory/branches/some_branch/@HEAD In this scenario, I got the following: svnInfo.url = https://<some IP address>/svn/REPO/directory/branches/some_branch url = https://<some IP address>/svn/REPO/directory/branches/some_branch/ The observable difference is that svnInfo.url has no trailing slash and url has a trailing slash. Therefore, I presume that the fix is to modify one of those two variables in some appropriate manner so that they agree with each other. If I have time to investigate the appropriate place to add (or remove) the slash, I'll report back.
          Hide
          laymanmb Matt Layman added a comment -

          I did some more research and have concluded that changing svnInfo.url to introduce a slash when present would be very difficult (even though it would be more correct since including a trailing slash in a URL is perfectly valid syntax).

          The problem is that the SVNInfo object is constructed with a File object. I ran a test and confirmed that none of the string getters from java.io.File will return a path with a trailing slash even if the trailing slash is provided in the File constructor. Thus, the SVNInfo object has no way of knowing if a user provided the trailing slash in a path or not.

          This means that the only feasible way to fix this is either to:

          1. Change ModuleLocation.getURL() so that it won't return a trailing slash (which could have unforseen impacts to other code which might depend on the trailing slash). OR...
          2. Change UpdateUpdater.isUpdatable so that when getURL is called, a trailing slash is removed if it exists, in order to normalize url and svnInfo.url.

          By the way, for those looking for a workaround, remove the trailing slash from the URL in your job. This has the effect of removing it from url so that it will match svnInfo.url. My earlier example would look like:

          https://<some IP address>/svn/REPO/directory/branches/some_branch@HEAD

          Of course, all of this may not be the only reason that url and svnInfo.url wouldn't match, but it is definitely one reason.

          Show
          laymanmb Matt Layman added a comment - I did some more research and have concluded that changing svnInfo.url to introduce a slash when present would be very difficult (even though it would be more correct since including a trailing slash in a URL is perfectly valid syntax). The problem is that the SVNInfo object is constructed with a File object. I ran a test and confirmed that none of the string getters from java.io.File will return a path with a trailing slash even if the trailing slash is provided in the File constructor. Thus, the SVNInfo object has no way of knowing if a user provided the trailing slash in a path or not. This means that the only feasible way to fix this is either to: Change ModuleLocation.getURL() so that it won't return a trailing slash (which could have unforseen impacts to other code which might depend on the trailing slash). OR... Change UpdateUpdater.isUpdatable so that when getURL is called, a trailing slash is removed if it exists, in order to normalize url and svnInfo.url . By the way, for those looking for a workaround , remove the trailing slash from the URL in your job. This has the effect of removing it from url so that it will match svnInfo.url . My earlier example would look like: https://<some IP address>/svn/REPO/directory/branches/some_branch@HEAD Of course, all of this may not be the only reason that url and svnInfo.url wouldn't match, but it is definitely one reason.
          Hide
          hetu Pjotr Vladrisk added a comment -

          From what I see, the comments so far describe two problems.

          1. multiple checkouts onto the workspace collide
          2. svn URLs do not match internally when ending on a slash

          Personally, I had problem No. 1. Jeff Payne's solution worked for that. Also, that problem yielded the very same error message as given in the description section above, while problem No. 2 seems to give a different error message under circumstances.

          Show
          hetu Pjotr Vladrisk added a comment - From what I see, the comments so far describe two problems. multiple checkouts onto the workspace collide svn URLs do not match internally when ending on a slash Personally, I had problem No. 1. Jeff Payne's solution worked for that. Also, that problem yielded the very same error message as given in the description section above, while problem No. 2 seems to give a different error message under circumstances.
          Hide
          alanbirtles Alan Birtles added a comment -

          just had a similar issue, in our case we had the svn url defined as http://server:80/svn/repo, I think the svn checkout strips out the port number then the polling thinks that the workspace doesn't contain the same working copy.

          Show
          alanbirtles Alan Birtles added a comment - just had a similar issue, in our case we had the svn url defined as http://server:80/svn/repo , I think the svn checkout strips out the port number then the polling thinks that the workspace doesn't contain the same working copy.
          Hide
          alghe Alexandru Gheorghe added a comment - - edited

          Same issue as Damien Finck described we're encoutering with latest Jenkins version to this date: v1.602:

          1. subversion plugin: 2.5
          2. multiple repositories:
            • some fetched to different folders
            • the rest fetched to one folder
          3. each time there's a trigger (timed at 15 minutes) even if there are no changes polling takes place
          4. the checkout will delete files and add ones on the first poll of the first repository, then the second does the same; what is interesting is that in the end it will balance out and it'll work but each time the build is triggered it will fetch the files in the workspace
          5. at each 15 minutes a new build takes 11MB so that in a couple of days we got over 1GB of disk space occupied; not something wanted

          Tried to append @HEAD at the URL but no luck:

          https://url/svn/branch@HEAD

          , it will indeed go to HEAD and poll there (unlike before where the revision was the date at which it was started) but still with "No changes" in the subversion polling log it still gets the files and checks out as if it goes back 1 revision and sees new changes.

          Show
          alghe Alexandru Gheorghe added a comment - - edited Same issue as Damien Finck described we're encoutering with latest Jenkins version to this date: v1.602 : subversion plugin: 2.5 multiple repositories: some fetched to different folders the rest fetched to one folder each time there's a trigger (timed at 15 minutes) even if there are no changes polling takes place the checkout will delete files and add ones on the first poll of the first repository, then the second does the same; what is interesting is that in the end it will balance out and it'll work but each time the build is triggered it will fetch the files in the workspace at each 15 minutes a new build takes 11MB so that in a couple of days we got over 1GB of disk space occupied; not something wanted Tried to append @HEAD at the URL but no luck: https://url/svn/branch@HEAD , it will indeed go to HEAD and poll there (unlike before where the revision was the date at which it was started) but still with "No changes" in the subversion polling log it still gets the files and checks out as if it goes back 1 revision and sees new changes.

            People

            • Assignee:
              Unassigned
              Reporter:
              giles Giles Dermody
            • Votes:
              15 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated: