Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-22853

SEVERE: Trying to unexport an object that's already unexported

    Details

    • Similar Issues:

      Description

      Just upgraded to 1.561, and I'm happy to see JENKINS-20769 fixed, but now I'm seeing these instead:

      SEVERE: Trying to unexport an object that's already unexported
      java.lang.IllegalStateException: Invalid object ID 6 iota=36
      	at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:277)
      	at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:300)
      	at hudson.remoting.Channel.unexport(Channel.java:600)
      	at hudson.remoting.UnexportCommand.execute(UnexportCommand.java:38)
      	at hudson.remoting.Channel$2.handle(Channel.java:475)
      	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60)
      Caused by: java.lang.Exception: Object was recently deallocated
      #6 (ref.0) : hudson.remoting.FastPipedOutputStream
        Created at Fri May 02 09:50:35 PDT 2014
      	at hudson.remoting.ExportTable$Entry.<init>(ExportTable.java:86)
      	at hudson.remoting.ExportTable.export(ExportTable.java:239)
      	at hudson.remoting.Channel.export(Channel.java:592)
      	at hudson.remoting.Pipe.writeObject(Pipe.java:158)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
      	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1495)
      	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
      	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
      	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
      	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
      	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
      	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
      	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
      	at hudson.remoting.UserRequest._serialize(UserRequest.java:155)
      	at hudson.remoting.UserRequest.serialize(UserRequest.java:164)
      	at hudson.remoting.UserRequest.<init>(UserRequest.java:62)
      	at hudson.remoting.Channel.callAsync(Channel.java:766)
      	at hudson.FilePath.read(FilePath.java:1640)
      	at hudson.FilePath.readToString(FilePath.java:1662)
      	at hudson.tools.JDKInstaller.performInstallation(JDKInstaller.java:126)
      	at hudson.tools.InstallerTranslator.getToolHome(InstallerTranslator.java:61)
      	at hudson.tools.ToolLocationNodeProperty.getToolHome(ToolLocationNodeProperty.java:107)
      	at hudson.tools.ToolInstallation.translateFor(ToolInstallation.java:204)
      	at hudson.model.JDK.forNode(JDK.java:126)
      	at org.jenkinsci.plugins.envinject.service.EnvInjectVariableGetter.getBuildVariables(EnvInjectVariableGetter.java:79)
      	at org.jenkinsci.plugins.envinject.EnvInjectListener.setUpEnvironmentWithoutJobPropertyObject(EnvInjectListener.java:231)
      	at org.jenkinsci.plugins.envinject.EnvInjectListener.setUpEnvironment(EnvInjectListener.java:46)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.createLauncher(AbstractBuild.java:570)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:476)
      	at hudson.model.Run.execute(Run.java:1709)
      	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      	at hudson.model.ResourceController.execute(ResourceController.java:88)
      	at hudson.model.Executor.run(Executor.java:231)
        Released at Fri May 02 09:50:35 PDT 2014
      	at hudson.remoting.ExportTable$Entry.release(ExportTable.java:115)
      	at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:303)
      	at hudson.remoting.Channel.unexport(Channel.java:600)
      	at hudson.remoting.ProxyOutputStream$EOF$1.run(ProxyOutputStream.java:384)
      	at hudson.remoting.PipeWriter$1.run(PipeWriter.java:158)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:111)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:744)
      
      	at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:270)
      	... 5 more
      Caused by:   Released at Fri May 02 09:50:35 PDT 2014
      	at hudson.remoting.ExportTable$Entry.release(ExportTable.java:115)
      	at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:303)
      	at hudson.remoting.Channel.unexport(Channel.java:600)
      	at hudson.remoting.ProxyOutputStream$EOF$1.run(ProxyOutputStream.java:384)
      	at hudson.remoting.PipeWriter$1.run(PipeWriter.java:158)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:111)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:744)
      

        Attachments

          Issue Links

            Activity

            Hide
            bbonn bbonn added a comment -

            HI There,

            We are still seeing this on Jenkins ver. 2.7.1. Is there additional things we can do to troubleshoot. We have also been crashing with permgen errors since upgrade, not sure if related. Please advise if i should reopen this issue. Thanks!

            SEVERE: Trying to unexport an object that's already unexported
            java.lang.IllegalStateException: Invalid object ID 412 iota=773
            at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:386)
            at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:409)
            at hudson.remoting.Channel.unexport(Channel.java:641)
            at hudson.remoting.UnexportCommand.execute(UnexportCommand.java:43)
            at hudson.remoting.Channel$1.handle(Channel.java:501)
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60)
            Caused by: java.lang.Exception: Object was recently deallocated
            #412 (ref.0) : object=null type=hudson.remoting.FastPipedOutputStream interfaces=[java.lang.Object]
            Created at Thu Feb 02 09:24:16 EST 2017
            at hudson.remoting.ExportTable$Entry.<init>(ExportTable.java:99)
            at hudson.remoting.ExportTable.export(ExportTable.java:305)
            at hudson.remoting.Channel.internalExport(Channel.java:629)
            at hudson.remoting.Pipe.writeObject(Pipe.java:158)
            at sun.reflect.GeneratedMethodAccessor1141.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
            at java.lang.reflect.Method.invoke(Unknown Source)
            at java.io.ObjectStreamClass.invokeWriteObject(Unknown Source)
            at java.io.ObjectOutputStream.writeSerialData(Unknown Source)
            at java.io.ObjectOutputStream.writeOrdinaryObject(Unknown Source)
            at java.io.ObjectOutputStream.writeObject0(Unknown Source)
            at java.io.ObjectOutputStream.defaultWriteFields(Unknown Source)
            at java.io.ObjectOutputStream.writeSerialData(Unknown Source)
            at java.io.ObjectOutputStream.writeOrdinaryObject(Unknown Source)
            at java.io.ObjectOutputStream.writeObject0(Unknown Source)
            at java.io.ObjectOutputStream.defaultWriteFields(Unknown Source)
            at java.io.ObjectOutputStream.writeSerialData(Unknown Source)
            at java.io.ObjectOutputStream.writeOrdinaryObject(Unknown Source)
            at java.io.ObjectOutputStream.writeObject0(Unknown Source)
            at java.io.ObjectOutputStream.writeObject(Unknown Source)
            at hudson.remoting.UserRequest._serialize(UserRequest.java:189)
            at hudson.remoting.UserRequest.serialize(UserRequest.java:198)
            at hudson.remoting.UserRequest.<init>(UserRequest.java:64)
            at hudson.remoting.Channel.callAsync(Channel.java:807)
            at hudson.FilePath.actAsync(FilePath.java:1087)
            at hudson.FilePath.read(FilePath.java:1782)
            at hudson.plugins.groovy.FileScriptSource.getScriptStream(FileScriptSource.java:52)
            at hudson.plugins.groovy.SystemGroovy.perform(SystemGroovy.java:93)
            at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
            at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
            at hudson.model.Build$BuildExecution.build(Build.java:205)
            at hudson.model.Build$BuildExecution.doRun(Build.java:162)
            at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534)
            at hudson.model.Run.execute(Run.java:1741)
            at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
            at hudson.model.ResourceController.execute(ResourceController.java:98)
            at hudson.model.Executor.run(Executor.java:410)
            Released at Thu Feb 02 09:24:16 EST 2017
            at hudson.remoting.ExportTable$Entry.release(ExportTable.java:131)
            at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:414)
            at hudson.remoting.Channel.unexport(Channel.java:641)
            at hudson.remoting.ProxyOutputStream$EOF$1.run(ProxyOutputStream.java:366)
            at hudson.remoting.PipeWriter$1.run(PipeWriter.java:158)
            at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
            at java.util.concurrent.FutureTask.run(Unknown Source)
            at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
            at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
            at org.jenkinsci.remoting.CallableDecorator.call(CallableDecorator.java:18)
            at hudson.remoting.CallableDecoratorList$1.call(CallableDecoratorList.java:21)
            at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
            at java.util.concurrent.FutureTask.run(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)
            Caused by: Command ProxyOutputStream.EOF(412) created at
            at hudson.remoting.Command.<init>(Command.java:67)
            at hudson.remoting.Command.<init>(Command.java:50)
            at hudson.remoting.ProxyOutputStream$EOF.<init>(ProxyOutputStream.java:358)
            at hudson.remoting.ProxyOutputStream.doClose(ProxyOutputStream.java:177)
            at hudson.remoting.ProxyOutputStream.close(ProxyOutputStream.java:173)
            at org.apache.commons.io.IOUtils.closeQuietly(IOUtils.java:303)
            at org.apache.commons.io.IOUtils.closeQuietly(IOUtils.java:274)
            at hudson.FilePath$33.invoke(FilePath.java:1795)
            at hudson.FilePath$33.invoke(FilePath.java:1782)
            at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2772)
            at hudson.remoting.UserRequest.perform(UserRequest.java:118)
            at hudson.remoting.UserRequest.perform(UserRequest.java:48)
            at hudson.remoting.Request$2.run(Request.java:328)
            at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
            at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
            at java.util.concurrent.FutureTask.run(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at hudson.remoting.Engine$1$1.run(Engine.java:63)
            ... 1 more
            at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:379)

            Show
            bbonn bbonn added a comment - HI There, We are still seeing this on Jenkins ver. 2.7.1. Is there additional things we can do to troubleshoot. We have also been crashing with permgen errors since upgrade, not sure if related. Please advise if i should reopen this issue. Thanks! SEVERE: Trying to unexport an object that's already unexported java.lang.IllegalStateException: Invalid object ID 412 iota=773 at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:386) at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:409) at hudson.remoting.Channel.unexport(Channel.java:641) at hudson.remoting.UnexportCommand.execute(UnexportCommand.java:43) at hudson.remoting.Channel$1.handle(Channel.java:501) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60) Caused by: java.lang.Exception: Object was recently deallocated #412 (ref.0) : object=null type=hudson.remoting.FastPipedOutputStream interfaces= [java.lang.Object] Created at Thu Feb 02 09:24:16 EST 2017 at hudson.remoting.ExportTable$Entry.<init>(ExportTable.java:99) at hudson.remoting.ExportTable.export(ExportTable.java:305) at hudson.remoting.Channel.internalExport(Channel.java:629) at hudson.remoting.Pipe.writeObject(Pipe.java:158) at sun.reflect.GeneratedMethodAccessor1141.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at java.io.ObjectStreamClass.invokeWriteObject(Unknown Source) at java.io.ObjectOutputStream.writeSerialData(Unknown Source) at java.io.ObjectOutputStream.writeOrdinaryObject(Unknown Source) at java.io.ObjectOutputStream.writeObject0(Unknown Source) at java.io.ObjectOutputStream.defaultWriteFields(Unknown Source) at java.io.ObjectOutputStream.writeSerialData(Unknown Source) at java.io.ObjectOutputStream.writeOrdinaryObject(Unknown Source) at java.io.ObjectOutputStream.writeObject0(Unknown Source) at java.io.ObjectOutputStream.defaultWriteFields(Unknown Source) at java.io.ObjectOutputStream.writeSerialData(Unknown Source) at java.io.ObjectOutputStream.writeOrdinaryObject(Unknown Source) at java.io.ObjectOutputStream.writeObject0(Unknown Source) at java.io.ObjectOutputStream.writeObject(Unknown Source) at hudson.remoting.UserRequest._serialize(UserRequest.java:189) at hudson.remoting.UserRequest.serialize(UserRequest.java:198) at hudson.remoting.UserRequest.<init>(UserRequest.java:64) at hudson.remoting.Channel.callAsync(Channel.java:807) at hudson.FilePath.actAsync(FilePath.java:1087) at hudson.FilePath.read(FilePath.java:1782) at hudson.plugins.groovy.FileScriptSource.getScriptStream(FileScriptSource.java:52) at hudson.plugins.groovy.SystemGroovy.perform(SystemGroovy.java:93) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779) at hudson.model.Build$BuildExecution.build(Build.java:205) at hudson.model.Build$BuildExecution.doRun(Build.java:162) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534) at hudson.model.Run.execute(Run.java:1741) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:410) Released at Thu Feb 02 09:24:16 EST 2017 at hudson.remoting.ExportTable$Entry.release(ExportTable.java:131) at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:414) at hudson.remoting.Channel.unexport(Channel.java:641) at hudson.remoting.ProxyOutputStream$EOF$1.run(ProxyOutputStream.java:366) at hudson.remoting.PipeWriter$1.run(PipeWriter.java:158) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at org.jenkinsci.remoting.CallableDecorator.call(CallableDecorator.java:18) at hudson.remoting.CallableDecoratorList$1.call(CallableDecoratorList.java:21) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: Command ProxyOutputStream.EOF(412) created at at hudson.remoting.Command.<init>(Command.java:67) at hudson.remoting.Command.<init>(Command.java:50) at hudson.remoting.ProxyOutputStream$EOF.<init>(ProxyOutputStream.java:358) at hudson.remoting.ProxyOutputStream.doClose(ProxyOutputStream.java:177) at hudson.remoting.ProxyOutputStream.close(ProxyOutputStream.java:173) at org.apache.commons.io.IOUtils.closeQuietly(IOUtils.java:303) at org.apache.commons.io.IOUtils.closeQuietly(IOUtils.java:274) at hudson.FilePath$33.invoke(FilePath.java:1795) at hudson.FilePath$33.invoke(FilePath.java:1782) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2772) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:328) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:63) ... 1 more at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:379)
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Bertrand Roussel
            Path:
            client/pom.xml
            http://jenkins-ci.org/commit/swarm-plugin/8ad209aaafad5ec6256aa547702870da083ff6cd
            Log:
            Upgrade remoting API to 2.62

            Version 2.62 contains a fix for the following bugs:

            This occurs mostly with Java 8.

            The remoting version matches with Jenkins 2.18/2.19/2.19.1/2.19.2.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Bertrand Roussel Path: client/pom.xml http://jenkins-ci.org/commit/swarm-plugin/8ad209aaafad5ec6256aa547702870da083ff6cd Log: Upgrade remoting API to 2.62 Version 2.62 contains a fix for the following bugs: https://issues.jenkins-ci.org/browse/JENKINS-22853 https://issues.jenkins-ci.org/browse/JENKINS-23271 (possibly) This occurs mostly with Java 8. The remoting version matches with Jenkins 2.18/2.19/2.19.1/2.19.2.
            Hide
            olivergondza Oliver Gondža added a comment -

            This fix will be part of 2.19.1 LTS naturally.

            Show
            olivergondza Oliver Gondža added a comment - This fix will be part of 2.19.1 LTS naturally.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            The fix has been release in remoting-2.62 and jenkins-2.17

            Show
            oleg_nenashev Oleg Nenashev added a comment - The fix has been release in remoting-2.62 and jenkins-2.17
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            From what I see, the issue happens because Jenkins tries to unexport ProxyInputStream objects from two sides.

            My repro steps:
            1) Create a test slave with 20 executors
            2) Run the Pipeline script below
            3) Interrupt build once it starts performing short stashes

            // The map we'll store the parallel steps in before executing hem.
            def stepsForParallel = [:]
            
            stage "Prepare stashed file"
            node("test") {
                sh "cp /Users/nenashev/Documents/jenkins/demo/jenkins-2.7.1/jenkins.war ."
                for (int i=0; i<10; i++) {
                    sh "cp jenkins.war jenkins_${i}.war"
                }
                writeFile file: 'foo.txt', text: 'Hello'
                stash includes: 'foo.txt', name: 'foo_stashed'
                stash includes: '*', name: 'jenkins_stashed'
            }
            
            
            stage "Parallel unstash"
            for (int i = 0; i < 19; i++) {
                def step = {
                    node("test") {
                        ws("ws_$i") {
                            unstash includes: '*', name: 'jenkins_stashed'
                        }
                    }
                }
                stepsForParallel["unstash_$i"] = step
            }
                
            stepsForParallel["fast_unstash"] = {
                node("test") {
                    ws("ws_fast") {
                        for (int i=0; i<2000; i++) {
                            unstash includes: '*', name: 'foo_stashed'
                        }
                    }
                }
            }
            
            parallel stepsForParallel
            

            Side 1. Remote call releases exports at the end of the call. E.g. if it gets interrupted

            Released at Thu Aug 04 13:12:16 MSK 2016
                	at hudson.remoting.ExportTable$Entry.release(ExportTable.java:131)
                	at hudson.remoting.ExportTable$ExportList.release(ExportTable.java:247)
                	at hudson.remoting.UserRequest.releaseExports(UserRequest.java:224)
                	at hudson.remoting.UserRequest.releaseExports(UserRequest.java:220)
                	at hudson.remoting.Channel.call(Channel.java:798)
                	at hudson.FilePath.act(FilePath.java:1007)
                	at hudson.FilePath.act(FilePath.java:996)
                	at hudson.FilePath.untarFrom(FilePath.java:728)
                	at org.jenkinsci.plugins.workflow.flow.StashManager.unstash(StashManager.java:114)
            

            Side 2. When ProxyInputStream gets closed, it sends asynchronous EOF command to the remote side. This EOF also tries to release the allocated object (FileInputStream in my case). By the time the request gets to the remote executor, UserRequest.releaseExports() unexports the object.

            ProxyInputStream is nested into RemoteInputStream, which is actually being used for many cases like stdin/stdout propagation in remote calls. So technically any remote call operation may overlap with UserRequest termination handler and cause an overlap in such case.

            So the solution would be to either make ProxyInputStream EOF command synchronous or to make it tolerant against race conditions.

            Reproducing the issue in tests

            Show
            oleg_nenashev Oleg Nenashev added a comment - From what I see, the issue happens because Jenkins tries to unexport ProxyInputStream objects from two sides. My repro steps: 1) Create a test slave with 20 executors 2) Run the Pipeline script below 3) Interrupt build once it starts performing short stashes // The map we'll store the parallel steps in before executing hem. def stepsForParallel = [:] stage "Prepare stashed file" node("test") { sh "cp /Users/nenashev/Documents/jenkins/demo/jenkins-2.7.1/jenkins.war ." for (int i=0; i<10; i++) { sh "cp jenkins.war jenkins_${i}.war" } writeFile file: 'foo.txt', text: 'Hello' stash includes: 'foo.txt', name: 'foo_stashed' stash includes: '*', name: 'jenkins_stashed' } stage "Parallel unstash" for (int i = 0; i < 19; i++) { def step = { node("test") { ws("ws_$i") { unstash includes: '*', name: 'jenkins_stashed' } } } stepsForParallel["unstash_$i"] = step } stepsForParallel["fast_unstash"] = { node("test") { ws("ws_fast") { for (int i=0; i<2000; i++) { unstash includes: '*', name: 'foo_stashed' } } } } parallel stepsForParallel Side 1. Remote call releases exports at the end of the call. E.g. if it gets interrupted Released at Thu Aug 04 13:12:16 MSK 2016 at hudson.remoting.ExportTable$Entry.release(ExportTable.java:131) at hudson.remoting.ExportTable$ExportList.release(ExportTable.java:247) at hudson.remoting.UserRequest.releaseExports(UserRequest.java:224) at hudson.remoting.UserRequest.releaseExports(UserRequest.java:220) at hudson.remoting.Channel.call(Channel.java:798) at hudson.FilePath.act(FilePath.java:1007) at hudson.FilePath.act(FilePath.java:996) at hudson.FilePath.untarFrom(FilePath.java:728) at org.jenkinsci.plugins.workflow.flow.StashManager.unstash(StashManager.java:114) Side 2. When ProxyInputStream gets closed, it sends asynchronous EOF command to the remote side. This EOF also tries to release the allocated object (FileInputStream in my case). By the time the request gets to the remote executor, UserRequest.releaseExports() unexports the object. ProxyInputStream is nested into RemoteInputStream, which is actually being used for many cases like stdin/stdout propagation in remote calls. So technically any remote call operation may overlap with UserRequest termination handler and cause an overlap in such case. So the solution would be to either make ProxyInputStream EOF command synchronous or to make it tolerant against race conditions. Reproducing the issue in tests

              People

              • Assignee:
                oleg_nenashev Oleg Nenashev
                Reporter:
                cg Christian Goetze
              • Votes:
                62 Vote for this issue
                Watchers:
                87 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: