Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-18924

Unusable File Structure under Windows for Configuration Backup

    Details

    • Type: Improvement
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Not A Defect
    • Component/s: other
    • Labels:
      None
    • Environment:
      Windows 7 64 Bit / WebSphere 7
    • Similar Issues:

      Description

      Well, our problem is as follows:

      Our .jenkins directory lies on a drive on Windows. We just want to make a backup of the config with the following result:
      -) we can't copy the folder for backup reasons (without errors -> changes the result)
      -) we can't zip the folder for backup reasons
      -) we can't bring the folder under version control (just testet Git)

      The reason seems to be the linking of files and directories under Windows. We may get arount this somehow, but generally using links seems to be not a good solution - storing relative paths in a text file or something like that would cause much less trouble regarding a configuration backup.

        Attachments

          Activity

          Hide
          kevin_brotcke Kevin Brotcke added a comment -

          Backing up on Windows is possible using robocopy and the /sl option, which only copies the symlink. Although, I haven't been able to find a good solution for compressing the backup. 7-zip treats the symlinks as regular folders creating a much larger backup than necessary.

          Show
          kevin_brotcke Kevin Brotcke added a comment - Backing up on Windows is possible using robocopy and the /sl option, which only copies the symlink. Although, I haven't been able to find a good solution for compressing the backup. 7-zip treats the symlinks as regular folders creating a much larger backup than necessary.
          Hide
          oleg_nenashev Oleg Nenashev added a comment -

          Have you tried Periodic Backup Plugin or other such stuff?
          Plugins know how to handle symlinks.

          Show
          oleg_nenashev Oleg Nenashev added a comment - Have you tried Periodic Backup Plugin or other such stuff? Plugins know how to handle symlinks.
          Hide
          rickybobpat Rick Patterson added a comment -

          @Oleg: Hi. I am trying to use the Periodoc Backup Plugin, and it suffers from an issue noted here - it creates a .zip file, but when the .zip file is restored, it has followed the symlinks for lastBuild, lastStableBuild, etc to the associated directories, and these former symlinks are restored a real directories, not symlinks. Subsequent builds shows errors in the logs as they cannot update what shoul dbe symlinks instead of directories. If I just use Windows explorer and copy the .jenkins structure to a new disk, as a back up, then during the Windows copy, it refuses to copy the symlink files, so the symlinks are missing in this case (and the next build creates them). I am just noting these results here, I am mot wishing to re-open this particular Jenkins issue (improvement )here, since I belive it to be working as designed - Jenkins has chosen to use these symlinks, and it is up to any tools such as backup to work correctly with the implementation. I will be opening this issue against the Periodic Backup plugin. Until that is fixed, I will look into the robocopy tool Kevin has suggested.

          Show
          rickybobpat Rick Patterson added a comment - @Oleg: Hi. I am trying to use the Periodoc Backup Plugin, and it suffers from an issue noted here - it creates a .zip file, but when the .zip file is restored, it has followed the symlinks for lastBuild, lastStableBuild, etc to the associated directories, and these former symlinks are restored a real directories, not symlinks. Subsequent builds shows errors in the logs as they cannot update what shoul dbe symlinks instead of directories. If I just use Windows explorer and copy the .jenkins structure to a new disk, as a back up, then during the Windows copy, it refuses to copy the symlink files, so the symlinks are missing in this case (and the next build creates them). I am just noting these results here, I am mot wishing to re-open this particular Jenkins issue (improvement )here, since I belive it to be working as designed - Jenkins has chosen to use these symlinks, and it is up to any tools such as backup to work correctly with the implementation. I will be opening this issue against the Periodic Backup plugin. Until that is fixed, I will look into the robocopy tool Kevin has suggested.
          Hide
          michaelheinzekeb Michael Heinze added a comment - - edited

          We are running Jenkins on Windows Server 2008 and we also had to deal with this issue.

          Our current solution is just to "tweak" the Windows system. In Windows Server 2008 you can withdraw the permission to create symbolic links for certain users and what we do is to withdraw this permission for the user the Tomcat service uses for its log-on. So, Jenkins just doesn't have the opportunity to create symlinks and creates plain text files instead. These can be zipped as usual.

          Newly there's another solution available: 7zip 9.33 alpha and onwards comes with a few new command line switches:

          • new -snh switch to store hard links as links (WIM and TAR formats only)
          • new -snl switch to store symbolic links as links (WIM and TAR formats only)

          Using these switches, it it possible to preserve the symlinks, and they will be restored during "unzip".
          The disadvantage of this is that - as mentioned above - these switches are only available for the TAR and WIM formats. These formats don't allow compression. So, you can backup the whole data in one file, preserving the symlinks, but it won't be compressed.

          See also the 7zip changelog: http://www.7-zip.org/history.txt

          Show
          michaelheinzekeb Michael Heinze added a comment - - edited We are running Jenkins on Windows Server 2008 and we also had to deal with this issue. Our current solution is just to "tweak" the Windows system. In Windows Server 2008 you can withdraw the permission to create symbolic links for certain users and what we do is to withdraw this permission for the user the Tomcat service uses for its log-on. So, Jenkins just doesn't have the opportunity to create symlinks and creates plain text files instead. These can be zipped as usual. Newly there's another solution available: 7zip 9.33 alpha and onwards comes with a few new command line switches: new -snh switch to store hard links as links (WIM and TAR formats only) new -snl switch to store symbolic links as links (WIM and TAR formats only) Using these switches, it it possible to preserve the symlinks, and they will be restored during "unzip". The disadvantage of this is that - as mentioned above - these switches are only available for the TAR and WIM formats. These formats don't allow compression. So, you can backup the whole data in one file, preserving the symlinks, but it won't be compressed. See also the 7zip changelog: http://www.7-zip.org/history.txt
          Hide
          rickybobpat Rick Patterson added a comment -

          Than for you comment, Michael, I had neglected to update this task regarding how we also resolved it. After my query above, we also resolved the issue in a similar way using cygwin tar. Using a cygwin tar at version 1.7.19 (which supports Windows symlinks), then a compressed tarfile is built, and released to a backup location (via jenkins job). If these tarballs are extracted using a cygwin tar of at least the same version 1.7.19, running under administrator account (which is required to created symlinks!), and having the following environment variable set: CYGWIN=winsymlinks:native, then the symlinks can be restored successfully.

          Show
          rickybobpat Rick Patterson added a comment - Than for you comment, Michael, I had neglected to update this task regarding how we also resolved it. After my query above, we also resolved the issue in a similar way using cygwin tar. Using a cygwin tar at version 1.7.19 (which supports Windows symlinks), then a compressed tarfile is built, and released to a backup location (via jenkins job). If these tarballs are extracted using a cygwin tar of at least the same version 1.7.19, running under administrator account (which is required to created symlinks!), and having the following environment variable set: CYGWIN=winsymlinks:native, then the symlinks can be restored successfully.

            People

            • Assignee:
              Unassigned
              Reporter:
              pueffl Wolfgang Grossinger
            • Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: