Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56249

userContent *zip* (all files in zip) stopped working at 2.164

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: core
    • Labels:
    • Environment:
      Windows 2008 R2 Enterprise, Jenkins 2.164+
    • Similar Issues:

      Description

      When using the (all files in zip) functionality starting at 2.164, either from the UI or URL (https:/myjenkins/userContent/*zip*/image.zip), the resulting image.zip has no content. 

      This has silently broken our builds, which utilize this functionality to retrieve certain content. Downgrading jenkins.war to 2.163 restores the functionality.

        Attachments

          Activity

          Hide
          danielbeck Daniel Beck added a comment -
          Show
          danielbeck Daniel Beck added a comment - Wadeck Follonier FYI
          Hide
          wfollonier Wadeck Follonier added a comment -

          Glenn Herbert Thank you for the report.

          I will not have the time to investigate this case further this week.

          Could you try using the WAR provided in JENKINS-56114 ? As they were also using Windows Server, that could the same root cause.

          In addition, did you try with the escape hatch: java.exe -Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true -jar jenkins.war ? (in both your current version and in the WAR provided in the other ticket)

          Then could you give me more information about your configuration, like are you using Docker inside your Windows Server, or just running java -jar ..., or using the service?

          If you can provide also additional information from the log of the build, that could be useful. Also, what is the content of the userContent in your instance (in terms of type, not the real files, but to know if there are symlinks, what are their targets, or just image files?)

          Thank you in advance for your next answer that will help me helping you

          Show
          wfollonier Wadeck Follonier added a comment - Glenn Herbert Thank you for the report. I will not have the time to investigate this case further this week. Could you try using the WAR provided in JENKINS-56114 ? As they were also using Windows Server, that could the same root cause. In addition, did you try with the escape hatch: java.exe -Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true -jar jenkins.war ? (in both your current version and in the WAR provided in the other ticket) Then could you give me more information about your configuration, like are you using Docker inside your Windows Server, or just running java -jar ... , or using the service? If you can provide also additional information from the log of the build, that could be useful. Also, what is the content of the userContent in your instance (in terms of type, not the real files, but to know if there are symlinks, what are their targets, or just image files?) Thank you in advance for your next answer that will help me helping you
          Hide
          kansasmann Glenn Herbert added a comment - - edited

          Wadeck Follonier: I cannot try using the WAR from JENKINS-56114  as this is a production system.  I will try to bring up a new jenkins and try it.
          To answer your other questions:

          • we are using the escape hatch -Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true
          • we are NOT using docker, just running the jenkins windows service
          • userContent has a mix of files, folders, and directory junctions (symlinks)

          The build logs have no real information in them as it was a silent failure, because the zip was empty.  The gradle code used was

          def f = new File("$buildOutput/staging.zip")
          if (!f.exists()) {
               new URL('http://myJenkins:8080/userContent/archive/components/Staging/*zip*/staging.zip').withInputStream{ i -> f.withOutputStream{ it << i }}
          }
          copy
          Unknown macro: {     from zipTree(f.getAbsolutePath())      into("$buildOutput/config/") }
          f.delete()

          From the UI, this is the Staging folder (this is not the actual folder used but shows the problem):

          If i then click on the (all files in zip), the Staging.zip file gets generated but its only 1KB

          And if its opened with 7zip the file is empty

          If I dump the file using od -c i see no content

          Hope this additional information helps!

           

          Show
          kansasmann Glenn Herbert added a comment - - edited Wadeck Follonier: I cannot try using the WAR from JENKINS-56114   as this is a production system.  I will try to bring up a new jenkins and try it. To answer your other questions: we are using the escape hatch -Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true we are NOT using docker, just running the jenkins windows service userContent has a mix of files, folders, and directory junctions (symlinks) The build logs have no real information in them as it was a silent failure, because the zip was empty.  The gradle code used was def f = new File("$buildOutput/staging.zip") if (!f.exists()) {      new URL('http://myJenkins:8080/userContent/archive/components/Staging/* zip */staging.zip').withInputStream{ i -> f.withOutputStream{ it << i }} } copy Unknown macro: {     from zipTree(f.getAbsolutePath())      into("$buildOutput/config/") } f.delete() From the UI, this is the Staging folder (this is not the actual folder used but shows the problem): If i then click on the (all files in zip), the Staging.zip file gets generated but its only 1KB And if its opened with 7zip the file is empty If I dump the file using od -c i see no content Hope this additional information helps!  
          Hide
          danielbeck Daniel Beck added a comment -

          Is this obsolete after recent changes in weekly releases?

          Show
          danielbeck Daniel Beck added a comment - Is this obsolete after recent changes in weekly releases?
          Hide
          kansasmann Glenn Herbert added a comment -

          The problem continues to persist even after upgrading all plugins and to the latest Jenkins version 2.168.

          The workaround we are using is to pre-zip the fetched content, to bypass the non-functional Jenkins zip functionality, so this is not as high a priority as it was.  However, it's still lost functionality that should be fixed.

          Show
          kansasmann Glenn Herbert added a comment - The problem continues to persist even after upgrading all plugins and to the latest Jenkins version 2.168. The workaround we are using is to pre-zip the fetched content, to bypass the non-functional Jenkins zip functionality, so this is not as high a priority as it was.  However, it's still lost functionality that should be fixed.
          Hide
          jvz Matt Sicker added a comment -

          Here is what I can reproduce which sound related to your problem:

          1. I run Jenkins with the -Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true flag set. (Or I modify it in the script console)
          2. In my userContent directory, I have one symbolic link to a file or directory outside of userContent such as ../../jobs/TestJob/lastSuccess/archive/
          3. I download the zip generated by the directory browser and find that it did not include those files despite using the flag.

          When I use Jenkins 2.163, the zip contains the data that I can also see in the browser. When I use Jenkins 2.164 or the latest release, the zip does not contain the files, though I can view the files in the browser still.

          Show
          jvz Matt Sicker added a comment - Here is what I can reproduce which sound related to your problem: I run Jenkins with the -Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true flag set. (Or I modify it in the script console) In my userContent directory, I have one symbolic link to a file or directory outside of userContent such as ../../jobs/TestJob/lastSuccess/archive/ I download the zip generated by the directory browser and find that it did not include those files despite using the flag. When I use Jenkins 2.163, the zip contains the data that I can also see in the browser. When I use Jenkins 2.164 or the latest release, the zip does not contain the files, though I can view the files in the browser still.
          Hide
          jvz Matt Sicker added a comment -

          Oh, and don't be confused by the double symbolic link thing going on there. This is still reproducible by simply linking to ../../config.xml

          Show
          jvz Matt Sicker added a comment - Oh, and don't be confused by the double symbolic link thing going on there. This is still reproducible by simply linking to ../../config.xml
          Hide
          kansasmann Glenn Herbert added a comment -

          Thanks Matt Sicker for verification.

          Show
          kansasmann Glenn Herbert added a comment - Thanks Matt Sicker for verification.
          Hide
          jvz Matt Sicker added a comment -

          Me or one of my teammates will be following up on this ticket to fix the problem. I believe I've been able to isolate the issue down to a difference in behavior when using the escape hatch where you can view files that are links from outside userContent or are themselves descendants of a link, but the same files are skipped in the zip file output regardless of the escape hatch.

          Show
          jvz Matt Sicker added a comment - Me or one of my teammates will be following up on this ticket to fix the problem. I believe I've been able to isolate the issue down to a difference in behavior when using the escape hatch where you can view files that are links from outside userContent or are themselves descendants of a link, but the same files are skipped in the zip file output regardless of the escape hatch.
          Hide
          arjo_poldervaart Arjo Poldervaart added a comment -

          Same issue here, we use symlinks a lot and are not able to upgrade at the moment to the latest LTS because of this issue. Is there a temporary workaround that can be made on a local Jenkins install? 

          Show
          arjo_poldervaart Arjo Poldervaart added a comment - Same issue here, we use symlinks a lot and are not able to upgrade at the moment to the latest LTS because of this issue. Is there a temporary workaround that can be made on a local Jenkins install? 
          Hide
          jthompson Jeff Thompson added a comment -

          Arjo Poldervaart, could you clarify some of your use cases for symlinks? How do you use them? Where do they link from and to? Do they link from inside a workspace to somewhere else on the system?

          Anyone else, please respond also, about your need for symlink support.

          I'm especially interested in how your symlink use works with zip files, as featured in the OP. Zip files generated by Jenkins do not contain symlinks but may resolve the symlink to the target and include all of that.

          One scenario, described in JENKINS-56114, involves a master or agent which resides in a path containing a symlink at the root or somewhere else higher in the path.

          Show
          jthompson Jeff Thompson added a comment - Arjo Poldervaart , could you clarify some of your use cases for symlinks? How do you use them? Where do they link from and to? Do they link from inside a workspace to somewhere else on the system? Anyone else, please respond also, about your need for symlink support. I'm especially interested in how your symlink use works with zip files, as featured in the OP. Zip files generated by Jenkins do not contain symlinks but may resolve the symlink to the target and include all of that. One scenario, described in JENKINS-56114 , involves a master or agent which resides in a path containing a symlink at the root or somewhere else higher in the path.
          Hide
          jthompson Jeff Thompson added a comment -

          And which of those use cases involve userContent, as described in the OP?

          Show
          jthompson Jeff Thompson added a comment - And which of those use cases involve userContent, as described in the OP?
          Hide
          kansasmann Glenn Herbert added a comment -

          Our use of (Windows) symlinks (directory junctions) is purely for directories. We store build artifacts on a corporate (remote) NAS drive (multi-TB), but use the jenkins userContent and its built-in file server as a way to 1) download artifacts via jenkins URL (ie curl http:/myJenkins/userContent/symlink/path/to/artifact), or 2) navigate to specific artifacts.  We don't ever request a zip on the actual symlinked folder but some content that resides within that symlinked folder tree.

           

          I would not expect a zip file to contain a symlink but rather its resolved target.

           

          Show
          kansasmann Glenn Herbert added a comment - Our use of (Windows) symlinks (directory junctions) is purely for directories. We store build artifacts on a corporate (remote) NAS drive (multi-TB), but use the jenkins userContent and its built-in file server as a way to 1) download artifacts via jenkins URL (ie curl http:/myJenkins/userContent/symlink/path/to/artifact), or 2) navigate to specific artifacts.  We don't ever request a zip on the actual symlinked folder but some content that resides within that symlinked folder tree.   I would not expect a zip file to contain a symlink but rather its resolved target.  
          Hide
          arjo_poldervaart Arjo Poldervaart added a comment - - edited

          Our use is similar to that of Glenn. To provide some context:

          If we want to release an artifact to production, we tag it in our SCM and let Jenkins build it, Jenkins then copies the artifact to a different location so we can download it at a later time (even if the workspace is cleared). The copy is done via a symlink created in the workspace by our build script (soft link on Linux) to a folder outside the workspace on disk.

          When we want to deploy the artifact we can simply download it via een link like: /view/assets/job/assets-BootstrapPortlet/ws/RELEASES/1.0.2/BootstrapPortletEAR.ear

          The _RELEASES_ folder __ is a symlink. So we basically use Jenkins as a file server just like Glenn.

           

          We do not use symlinks in the userContent folder, but we do use it in the job workspace.

          Show
          arjo_poldervaart Arjo Poldervaart added a comment - - edited Our use is similar to that of Glenn. To provide some context: If we want to release an artifact to production, we tag it in our SCM and let Jenkins build it, Jenkins then copies the artifact to a different location so we can download it at a later time (even if the workspace is cleared). The copy is done via a symlink created in the workspace by our build script (soft link on Linux) to a folder outside the workspace on disk. When we want to deploy the artifact we can simply download it via een link like: /view/assets/job/assets-BootstrapPortlet/ws/ RELEASES /1.0.2/BootstrapPortletEAR.ear The _ RELEASES _ folder __ is a symlink. So we basically use Jenkins as a file server just like Glenn.   We do not use symlinks in the userContent folder, but we do use it in the job workspace.
          Hide
          jthompson Jeff Thompson added a comment -

          Glenn Herbert and Arjo Poldervaart, thanks for your responses on describing your use models. That helps clarify some things.

          Have you read the security advisory from last year at https://jenkins.io/security/advisory/2018-12-05/ ? Have you tried disabling that fix via the system property described there? It sounds like your use cases are some of the rare situations where that might make sense.

          Show
          jthompson Jeff Thompson added a comment - Glenn Herbert and Arjo Poldervaart , thanks for your responses on describing your use models. That helps clarify some things. Have you read the security advisory from last year at  https://jenkins.io/security/advisory/2018-12-05/  ? Have you tried disabling that fix via the system property described there? It sounds like your use cases are some of the rare situations where that might make sense.
          Hide
          arjo_poldervaart Arjo Poldervaart added a comment -

          Hi Jeff,

          We have been running with "-Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true" for quite a while already. The setting does not resolve the current issue.

          Show
          arjo_poldervaart Arjo Poldervaart added a comment - Hi Jeff, We have been running with "-Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true" for quite a while already. The setting does not resolve the current issue.
          Hide
          kansasmann Glenn Herbert added a comment -

          Same here, been running with SymlinkEscape=true for quite a while.

          Show
          kansasmann Glenn Herbert added a comment - Same here, been running with SymlinkEscape=true for quite a while.
          Hide
          jthompson Jeff Thompson added a comment -

          I'm able to reproduce Matt's simple test case on Unix and Windows. Hopefully it's a reasonably good representation of the other problems people are experiencing. Looks like the `allowSymlinkEscape` property isn't working the way it was intended. I'll see if I can figure something out to make it work better. It may take a while to get that change put together.

          Show
          jthompson Jeff Thompson added a comment - I'm able to reproduce Matt's simple test case on Unix and Windows. Hopefully it's a reasonably good representation of the other problems people are experiencing. Looks like the `allowSymlinkEscape` property isn't working the way it was intended. I'll see if I can figure something out to make it work better. It may take a while to get that change put together.

            People

            • Assignee:
              Unassigned
              Reporter:
              kansasmann Glenn Herbert
            • Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated: