Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-26008

Use of compression during copying of artifacts cut throughput by a factor of 3

    Details

    • Similar Issues:
    • Released As:
      jenkins-2.196

      Description

      We have 19GB of tarballs and isos that are built on a Linux slave. The pulling of artifacts to the master takes 33 minutes over a local network. I modified the Jenkins source to not use compression and reran the job; the job took 9 minutes. This mirrors what happens if you use tar and ssh from the command line with the same machines.

      My method of test was to have a job in Jenkins that only archived the artifacts. I pre-populated the workspace with the data that I wanted archived.

      To turn of compression I modified the function copyRecursiveTo in FilePath.java (core/src/main/hudson). I change the 4 lines in that function that referenced TarCompression.GZIP to TarCompression.NONE for my test.

      I don't suggest just simply changing the TarCompression to NONE but to make it configurable. It should be configurable due to slaves running over the internet, remotely, or in the cloud, etc where compression may be desirable.

      As a note if I tar up the same data with compression and pipe it to ssh and untar it on the destination it takes 19 minutes. Doing the same thing without compression it takes 5 minutes.

        Attachments

          Issue Links

            Activity

            Hide
            tom_gl Thomas de Grenier de Latour added a comment -

            Just closed JENKINS-30815 as a duplicate of this one, after Carl George pointed it to me. But you can still have a look there for more evidence of how undesirable this compression can be in some cases.

            Show
            tom_gl Thomas de Grenier de Latour added a comment - Just closed JENKINS-30815 as a duplicate of this one, after Carl George pointed it to me. But you can still have a look there for more evidence of how undesirable this compression can be in some cases.
            Hide
            slonopotamusorama Marat Radchenko added a comment -

            Fixed in PR#4205 and released in Jenkins-2.196.

            Show
            slonopotamusorama Marat Radchenko added a comment - Fixed in PR#4205 and released in Jenkins-2.196.

              People

              • Assignee:
                Unassigned
                Reporter:
                carlg Carl George
              • Votes:
                5 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: