Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54252

unstash very slow blocked on SecureRandom

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not A Defect
    • Icon: Minor Minor
    • remoting
    • None

      unstash in pipeline jobs takes quite a long time and I see a decent number of BLOCKED threads with the following stack

      RemoteInputStream greedy pump thread: hudson.remoting.RemoteInputStream$Greedy
      at hudson.remoting.RemoteInputStream.<init>(RemoteInputStream.java:106)
      at hudson.remoting.RemoteInputStream.<init>(RemoteInputStream.java:89)
      at hudson.FilePath.untar(FilePath.java:569)
      at org.jenkinsci.plugins.workflow.flow.StashManager.unstash(StashManager.java:165)
      at org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:76)
      at org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:63)
      at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:50)
      at hudson.security.ACL.impersonate(ACL.java:290)
      at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:47)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
      
      java.security.SecureRandom.nextBytes(SecureRandom.java:468)
      sun.security.ssl.CipherBox.createExplicitNonce(CipherBox.java:1015)
      sun.security.ssl.EngineOutputRecord.write(EngineOutputRecord.java:287)
      sun.security.ssl.EngineOutputRecord.write(EngineOutputRecord.java:225)
      sun.security.ssl.EngineWriter.writeRecord(EngineWriter.java:186)
      sun.security.ssl.SSLEngineImpl.writeRecord(SSLEngineImpl.java:1300)
      sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1271)
      sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
      javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
      org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:491)
      org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248)
      org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200)
      org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doSend(ProtocolStack.java:692)
      org.jenkinsci.remoting.protocol.ApplicationLayer.write(ApplicationLayer.java:157)
      org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.write(ChannelApplicationLayer.java:299)
      hudson.remoting.AbstractByteBufferCommandTransport.write(AbstractByteBufferCommandTransport.java:304)
      hudson.remoting.Channel.send(Channel.java:721)
      hudson.remoting.ProxyOutputStream.write(ProxyOutputStream.java:144)
      hudson.remoting.RemoteInputStream$1.run(RemoteInputStream.java:148)
      

      Interestingly there is enough entropy available (as we have haveged installed)

      cat /proc/sys/kernel/random/entropy_avail
      3878
      

      The key unstash is happening in a parallel step (basically one stage creates an artifact around 1G and then ~50 steps which run in parallel unstash the artifact).

      This is on Linux and Jenkins LTS 2.138.2

      The Slaves which do the unstashing are on windows and use the jnlp remoting.

            jthompson Jeff Thompson
            tario Patrick Ruckstuhl
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: