Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54252

unstash very slow blocked on SecureRandom

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Minor
    • Resolution: Not A Defect
    • Component/s: remoting
    • Labels:
      None
    • Similar Issues:

      Description

      unstash in pipeline jobs takes quite a long time and I see a decent number of BLOCKED threads with the following stack

      RemoteInputStream greedy pump thread: hudson.remoting.RemoteInputStream$Greedy
      at hudson.remoting.RemoteInputStream.<init>(RemoteInputStream.java:106)
      at hudson.remoting.RemoteInputStream.<init>(RemoteInputStream.java:89)
      at hudson.FilePath.untar(FilePath.java:569)
      at org.jenkinsci.plugins.workflow.flow.StashManager.unstash(StashManager.java:165)
      at org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:76)
      at org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:63)
      at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:50)
      at hudson.security.ACL.impersonate(ACL.java:290)
      at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:47)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
      
      java.security.SecureRandom.nextBytes(SecureRandom.java:468)
      sun.security.ssl.CipherBox.createExplicitNonce(CipherBox.java:1015)
      sun.security.ssl.EngineOutputRecord.write(EngineOutputRecord.java:287)
      sun.security.ssl.EngineOutputRecord.write(EngineOutputRecord.java:225)
      sun.security.ssl.EngineWriter.writeRecord(EngineWriter.java:186)
      sun.security.ssl.SSLEngineImpl.writeRecord(SSLEngineImpl.java:1300)
      sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1271)
      sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
      javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
      org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:491)
      org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248)
      org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200)
      org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doSend(ProtocolStack.java:692)
      org.jenkinsci.remoting.protocol.ApplicationLayer.write(ApplicationLayer.java:157)
      org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.write(ChannelApplicationLayer.java:299)
      hudson.remoting.AbstractByteBufferCommandTransport.write(AbstractByteBufferCommandTransport.java:304)
      hudson.remoting.Channel.send(Channel.java:721)
      hudson.remoting.ProxyOutputStream.write(ProxyOutputStream.java:144)
      hudson.remoting.RemoteInputStream$1.run(RemoteInputStream.java:148)
      

      Interestingly there is enough entropy available (as we have haveged installed)

      cat /proc/sys/kernel/random/entropy_avail
      3878
      

      The key unstash is happening in a parallel step (basically one stage creates an artifact around 1G and then ~50 steps which run in parallel unstash the artifact).

      This is on Linux and Jenkins LTS 2.138.2

      The Slaves which do the unstashing are on windows and use the jnlp remoting.

        Attachments

          Activity

          Hide
          jthompson Jeff Thompson added a comment -

          See the documentation for Pipeline:stash: https://jenkins.io/doc/pipeline/steps/workflow-basic-steps/#stash-stash-some-files-to-be-used-later-in-the-build

          "Note that the stash and unstash steps are designed for use with small files. For large data transfers, use the External Workspace Manager plugin, or use an external repository manager such as Nexus or Artifactory. This is because stashed files are archived in a compressed TAR, and with large files this demands considerable on-master resources, particularly CPU time. There's not a hard stash size limit, but between 5-100 MB you should probably consider alternatives."

          Show
          jthompson Jeff Thompson added a comment - See the documentation for Pipeline:stash: https://jenkins.io/doc/pipeline/steps/workflow-basic-steps/#stash-stash-some-files-to-be-used-later-in-the-build " Note  that the  stash  and  unstash  steps are designed for use with small files. For large data transfers, use the External Workspace Manager plugin, or use an external repository manager such as Nexus or Artifactory. This is because stashed files are archived in a compressed TAR, and with large files this demands considerable on-master resources, particularly CPU time. There's not a hard stash size limit, but between 5-100 MB you should probably consider alternatives."

            People

            • Assignee:
              jthompson Jeff Thompson
              Reporter:
              tario Patrick Ruckstuhl
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: