Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38640

Unstash painfully slow for large artifacts (due to compression?)

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Minor
    • Resolution: Fixed
    • Component/s: workflow-api-plugin
    • Environment:
    • Similar Issues:

      Description

      When sending a stash across the network, the data transfer can be significantly slower than the underlying network. This seems to be due to some slow on-the-fly compression happening during the transfer. Using the following script, the Unstash phase takes 1-2 seconds for 10MB of /dev/zero, and 49 seconds for 10MB of /dev/urandom. The Stash phase consistently takes 1-2 seconds. This makes me think there's a compression step, since /dev/zero should compress much faster than /dev/urandom.

      node {
          deleteDir()
          stage "Stash"
          sh "dd if=/dev/zero of=data bs=1M count=10"
          stash name: 'build_outputs', includes: 'data'
          sh "date"
          node('lab') {
              deleteDir()
              stage "Unstash"
              unstash 'build_outputs'
              sh "ls -al"
          }
      }
      

      This looks similar to JENKINS-36914, in particular this comment:
      https://issues.jenkins-ci.org/browse/JENKINS-36914?focusedCommentId=268472&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-268472

      I'm not calling this a dupe of that because the original text refers to stash only, and this bug seems isolated to unstash. Also this is on x86, not ARM.

        Attachments

          Issue Links

            Activity

            Hide
            seanf Sean Flanigan added a comment -

            The External Workspace Manager plugin seems to require sharing drives between master and slaves, which rules it out in a lot of situations.

            If, instead of stashing, the official recommendation is to use an external repository manager such as Nexus or Artifactory, it would be good to have some pipeline examples. For instance, how to stash the majority of workspace files, whilst letting the repository manager handle the larger files, and still ensuring that the right files are used together throughout the pipeline.

            Show
            seanf Sean Flanigan added a comment - The External Workspace Manager plugin seems to require sharing drives between master and slaves, which rules it out in a lot of situations. If, instead of stashing, the official recommendation is to use an external repository manager such as Nexus or Artifactory, it would be good to have some pipeline examples. For instance, how to stash the majority of workspace files, whilst letting the repository manager handle the larger files, and still ensuring that the right files are used together throughout the pipeline.
            Hide
            jkoleszar John Koleszar added a comment -

            The admonition not to use stash is too strong, in my opinion. I can see plenty of situations where having a repository manager or setting up shared storage are completely unnecessary complexity. The latest release of this plugin performs as I'd expect.

            Show
            jkoleszar John Koleszar added a comment - The admonition not to use stash is too strong, in my opinion. I can see plenty of situations where having a repository manager or setting up shared storage are completely unnecessary complexity. The latest release of this plugin performs as I'd expect.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            There is NO official recommendation to avoid stash/unstash. I've just mentioned it as a possible workaround

            Show
            oleg_nenashev Oleg Nenashev added a comment - There is NO official recommendation to avoid stash/unstash. I've just mentioned it as a possible workaround
            Hide
            jkoleszar John Koleszar added a comment -

            Thanks, Oleg. This is the documentation I was referring to about stash only being suitable for small files:
            https://github.com/jenkinsci/workflow-basic-steps-plugin/commit/413df48bdcb832261e8fb110150eeb8069e77c33

            Maybe there's an upper limit where it falls over, but I haven't hit it yet. In my opinion, if there were one, it'd be a bug worth fixing, rather than something to document away as WAI.

            Show
            jkoleszar John Koleszar added a comment - Thanks, Oleg. This is the documentation I was referring to about stash only being suitable for small files: https://github.com/jenkinsci/workflow-basic-steps-plugin/commit/413df48bdcb832261e8fb110150eeb8069e77c33 Maybe there's an upper limit where it falls over, but I haven't hit it yet. In my opinion, if there were one, it'd be a bug worth fixing, rather than something to document away as WAI.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            The fix has been backported to 2.19.3

            Show
            oleg_nenashev Oleg Nenashev added a comment - The fix has been backported to 2.19.3

              People

              • Assignee:
                andresrc Andres Rodriguez
                Reporter:
                jkoleszar John Koleszar
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: