Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54252

unstash very slow blocked on SecureRandom

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Minor
    • Resolution: Not A Defect
    • Component/s: remoting
    • Labels:
      None
    • Similar Issues:

      Description

      unstash in pipeline jobs takes quite a long time and I see a decent number of BLOCKED threads with the following stack

      RemoteInputStream greedy pump thread: hudson.remoting.RemoteInputStream$Greedy
      at hudson.remoting.RemoteInputStream.<init>(RemoteInputStream.java:106)
      at hudson.remoting.RemoteInputStream.<init>(RemoteInputStream.java:89)
      at hudson.FilePath.untar(FilePath.java:569)
      at org.jenkinsci.plugins.workflow.flow.StashManager.unstash(StashManager.java:165)
      at org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:76)
      at org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:63)
      at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:50)
      at hudson.security.ACL.impersonate(ACL.java:290)
      at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:47)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
      
      java.security.SecureRandom.nextBytes(SecureRandom.java:468)
      sun.security.ssl.CipherBox.createExplicitNonce(CipherBox.java:1015)
      sun.security.ssl.EngineOutputRecord.write(EngineOutputRecord.java:287)
      sun.security.ssl.EngineOutputRecord.write(EngineOutputRecord.java:225)
      sun.security.ssl.EngineWriter.writeRecord(EngineWriter.java:186)
      sun.security.ssl.SSLEngineImpl.writeRecord(SSLEngineImpl.java:1300)
      sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1271)
      sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
      javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
      org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:491)
      org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248)
      org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200)
      org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doSend(ProtocolStack.java:692)
      org.jenkinsci.remoting.protocol.ApplicationLayer.write(ApplicationLayer.java:157)
      org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.write(ChannelApplicationLayer.java:299)
      hudson.remoting.AbstractByteBufferCommandTransport.write(AbstractByteBufferCommandTransport.java:304)
      hudson.remoting.Channel.send(Channel.java:721)
      hudson.remoting.ProxyOutputStream.write(ProxyOutputStream.java:144)
      hudson.remoting.RemoteInputStream$1.run(RemoteInputStream.java:148)
      

      Interestingly there is enough entropy available (as we have haveged installed)

      cat /proc/sys/kernel/random/entropy_avail
      3878
      

      The key unstash is happening in a parallel step (basically one stage creates an artifact around 1G and then ~50 steps which run in parallel unstash the artifact).

      This is on Linux and Jenkins LTS 2.138.2

      The Slaves which do the unstashing are on windows and use the jnlp remoting.

        Attachments

          Activity

            People

            • Assignee:
              jthompson Jeff Thompson
              Reporter:
              tario Patrick Ruckstuhl
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: