Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-20345

ZIP file download generates corrupt zip file

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      May be related to JENKINS-19752
      One of our jobs has a directory in its workspace containing about 30,000 files (small files < 100KB). A zip-download of this directory results in a zip file (size 12MB) that cannot be unzipped:
      "End-of-central-directory signature not found. Either this file is not a zipfile, or it constitutes one disk of a multi-part archive. In the latter case the central directory and zipfile comment will be found on the last disk(s) of this archive."

      Zipping this directory at the command prompt results in a valid, unzippable file with size 42MB.

      While analyzing this, I tried downloading the Jenkins created zip file a few times using wget and the browser. Most of the time wget aborted with error 500. Trying the download in the browser sometimes resulted in an IO error (Stack trace attached).

      While I was trying around, another job was started and failed with "Too many open files".

      So it seems to me that the zip download opens all the files when adding them to the zip but apparently they are not closed.

        Attachments

          Activity

          Hide
          jsternberg Jonathan Sternberg added a comment -

          We're also having this problem with a directory that has ~8,000 files. It seems to be related to this line of code: https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/model/DirectoryBrowserSupport.java#L354.

          Switching this to "copyStreamAndClose" should fix this issue.

          Show
          jsternberg Jonathan Sternberg added a comment - We're also having this problem with a directory that has ~8,000 files. It seems to be related to this line of code: https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/model/DirectoryBrowserSupport.java#L354 . Switching this to "copyStreamAndClose" should fix this issue.
          Hide
          jsternberg Jonathan Sternberg added a comment -

          I've sent a pull request that fixes this issue.

          https://github.com/jenkinsci/jenkins/pull/1089

          Show
          jsternberg Jonathan Sternberg added a comment - I've sent a pull request that fixes this issue. https://github.com/jenkinsci/jenkins/pull/1089
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Jonathan A. Sternberg
          Path:
          core/src/main/java/hudson/model/DirectoryBrowserSupport.java
          http://jenkins-ci.org/commit/jenkins/f0bbc66921fec56dd9565c2a5146734f20f2fb6c
          Log:
          [FIXED JENKINS-20345] file descriptor leak in zip file download

          When downloading a zip file with all of the artifacts, file descriptors
          would be opened and never closed. This resulted in many leaked file
          descriptors which would only be cleaned up the next time the garbage
          collector was run.

          For directories with a large number of files, the zip file would be
          corrupted because an exception was thrown during the process of creating
          the zip file when the process ran out of file descriptors. Jenkins was
          then left in a very brittle state as random threads wouldn't be able to
          open files or sockets anymore until the garbage collector was run.

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jonathan A. Sternberg Path: core/src/main/java/hudson/model/DirectoryBrowserSupport.java http://jenkins-ci.org/commit/jenkins/f0bbc66921fec56dd9565c2a5146734f20f2fb6c Log: [FIXED JENKINS-20345] file descriptor leak in zip file download When downloading a zip file with all of the artifacts, file descriptors would be opened and never closed. This resulted in many leaked file descriptors which would only be cleaned up the next time the garbage collector was run. For directories with a large number of files, the zip file would be corrupted because an exception was thrown during the process of creating the zip file when the process ran out of file descriptors. Jenkins was then left in a very brittle state as random threads wouldn't be able to open files or sockets anymore until the garbage collector was run.
          Hide
          dogfood dogfood added a comment -

          Integrated in jenkins_main_trunk #3168
          [FIXED JENKINS-20345] file descriptor leak in zip file download (Revision f0bbc66921fec56dd9565c2a5146734f20f2fb6c)

          Result = SUCCESS
          jonathansternberg : f0bbc66921fec56dd9565c2a5146734f20f2fb6c
          Files :

          • core/src/main/java/hudson/model/DirectoryBrowserSupport.java
          Show
          dogfood dogfood added a comment - Integrated in jenkins_main_trunk #3168 [FIXED JENKINS-20345] file descriptor leak in zip file download (Revision f0bbc66921fec56dd9565c2a5146734f20f2fb6c) Result = SUCCESS jonathansternberg : f0bbc66921fec56dd9565c2a5146734f20f2fb6c Files : core/src/main/java/hudson/model/DirectoryBrowserSupport.java
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Jonathan A. Sternberg
          Path:
          core/src/main/java/hudson/model/DirectoryBrowserSupport.java
          http://jenkins-ci.org/commit/jenkins/53e2f7e129f52f67f3efd93da4db2b6dabe84d69
          Log:
          [FIXED JENKINS-20345] file descriptor leak in zip file download

          When downloading a zip file with all of the artifacts, file descriptors
          would be opened and never closed. This resulted in many leaked file
          descriptors which would only be cleaned up the next time the garbage
          collector was run.

          For directories with a large number of files, the zip file would be
          corrupted because an exception was thrown during the process of creating
          the zip file when the process ran out of file descriptors. Jenkins was
          then left in a very brittle state as random threads wouldn't be able to
          open files or sockets anymore until the garbage collector was run.

          (cherry picked from commit f0bbc66921fec56dd9565c2a5146734f20f2fb6c)

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jonathan A. Sternberg Path: core/src/main/java/hudson/model/DirectoryBrowserSupport.java http://jenkins-ci.org/commit/jenkins/53e2f7e129f52f67f3efd93da4db2b6dabe84d69 Log: [FIXED JENKINS-20345] file descriptor leak in zip file download When downloading a zip file with all of the artifacts, file descriptors would be opened and never closed. This resulted in many leaked file descriptors which would only be cleaned up the next time the garbage collector was run. For directories with a large number of files, the zip file would be corrupted because an exception was thrown during the process of creating the zip file when the process ran out of file descriptors. Jenkins was then left in a very brittle state as random threads wouldn't be able to open files or sockets anymore until the garbage collector was run. (cherry picked from commit f0bbc66921fec56dd9565c2a5146734f20f2fb6c)

            People

            • Assignee:
              Unassigned
              Reporter:
              wolterhis Lars
            • Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: