Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-22734

OutOfMemory on file transfer in 1.560

    Details

    • Similar Issues:

      Description

      Reproduced in jenkins ver 1.560

      Apr 21, 2014 10:46:43 AM hudson.remoting.Channel$2 handle
      SEVERE: Failed to execute command Pipe.EOF(1895) (channel SRV6026)
      java.lang.NullPointerException
              at hudson.remoting.ProxyWriter$EOF.execute(ProxyWriter.java:177)
              at hudson.remoting.Channel$2.handle(Channel.java:461)
              at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60)
      
      Apr 21, 2014 10:46:43 AM hudson.remoting.Channel$2 handle
      SEVERE: This command is created here
      Command Pipe.EOF(1895) created at
              at hudson.remoting.Command.<init>(Command.java:67)
              at hudson.remoting.Command.<init>(Command.java:50)
              at hudson.remoting.ProxyWriter$EOF.<init>(ProxyWriter.java:169)
              at hudson.remoting.ProxyWriter.close(ProxyWriter.java:124)
              at hudson.remoting.ProxyWriter.finalize(ProxyWriter.java:132)
              at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
              at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:101)
              at java.lang.ref.Finalizer.access$100(Finalizer.java:32)
              at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:190)
      

      Issue appears when jenkins try to archive artifacts when job runs on slave. Some of artifacts has 150+ Mb size.
      After abort job in console I have:

      ERROR: Failed to parse POMs
      java.io.IOException: java.io.IOException: Failed to extract /var/lib/jenkins/workspace/Platform-0/assemblies/karaf/transfer of 3 files
      	at hudson.FilePath.readFromTar(FilePath.java:2095)
      	at hudson.FilePath.copyRecursiveTo(FilePath.java:2007)
      	at jenkins.model.StandardArtifactManager.archive(StandardArtifactManager.java:57)
      	at hudson.maven.MavenBuild$ProxyImpl.performArchiving(MavenBuild.java:474)
      	at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:841)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:519)
      	at hudson.model.Run.execute(Run.java:1703)
      	at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:525)
      	at hudson.model.ResourceController.execute(ResourceController.java:88)
      	at hudson.model.Executor.run(Executor.java:231)
      Caused by: java.io.IOException
      	at hudson.remoting.FastPipedInputStream.read(FastPipedInputStream.java:177)
      	at hudson.util.HeadBufferingStream.read(HeadBufferingStream.java:61)
      	at com.jcraft.jzlib.InflaterInputStream.fill(InflaterInputStream.java:175)
      	at com.jcraft.jzlib.InflaterInputStream.read(InflaterInputStream.java:106)
      	at org.apache.tools.tar.TarBuffer.readBlock(TarBuffer.java:257)
      	at org.apache.tools.tar.TarBuffer.readRecord(TarBuffer.java:223)
      	at hudson.org.apache.tools.tar.TarInputStream.read(TarInputStream.java:345)
      	at java.io.FilterInputStream.read(FilterInputStream.java:107)
      	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1792)
      	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769)
      	at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744)
      	at hudson.util.IOUtils.copy(IOUtils.java:40)
      	at hudson.FilePath.readFromTar(FilePath.java:2085)
      	... 9 more
      
      	at hudson.FilePath.copyRecursiveTo(FilePath.java:2014)
      	at jenkins.model.StandardArtifactManager.archive(StandardArtifactManager.java:57)
      	at hudson.maven.MavenBuild$ProxyImpl.performArchiving(MavenBuild.java:474)
      	at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:841)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:519)
      	at hudson.model.Run.execute(Run.java:1703)
      	at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:525)
      	at hudson.model.ResourceController.execute(ResourceController.java:88)
      	at hudson.model.Executor.run(Executor.java:231)
      Caused by: java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected reader termination
      	at hudson.remoting.Request$1.get(Request.java:278)
      	at hudson.remoting.Request$1.get(Request.java:210)
      	at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
      	at hudson.FilePath.copyRecursiveTo(FilePath.java:2010)
      	... 8 more
      Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected reader termination
      	at hudson.remoting.Request.abort(Request.java:299)
      	at hudson.remoting.Channel.terminate(Channel.java:801)
      	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:76)
      Caused by: java.io.IOException: Unexpected reader termination
      	... 1 more
      Caused by: java.lang.OutOfMemoryError: Java heap space
      	at java.util.Arrays.copyOf(Arrays.java:2271)
      	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
      	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
      	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
      	at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:87)
      	at hudson.remoting.ChunkedInputStream.read(ChunkedInputStream.java:46)
      	at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:88)
      	at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:33)
      	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
      	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
      Looks like the node went offline during the build. Check the slave log for the details.FATAL: /var/lib/jenkins/logs/slaves/SERVER6026/slave.log (No such file or directory)
      java.io.FileNotFoundException: /var/lib/jenkins/logs/slaves/SERVER6026/slave.log (No such file or directory)
      	at java.io.RandomAccessFile.open(Native Method)
      	at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
      	at org.kohsuke.stapler.framework.io.LargeText$FileSession.<init>(LargeText.java:397)
      	at org.kohsuke.stapler.framework.io.LargeText$2.open(LargeText.java:120)
      	at org.kohsuke.stapler.framework.io.LargeText.writeLogTo(LargeText.java:210)
      	at hudson.console.AnnotatedLargeText.writeHtmlTo(AnnotatedLargeText.java:156)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:532)
      	at hudson.model.Run.execute(Run.java:1703)
      	at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:525)
      	at hudson.model.ResourceController.execute(ResourceController.java:88)
      	at hudson.model.Executor.run(Executor.java:231)
      

      Jenkins slave runs with: -XX:MaxPermSize=256m -Xmx1024m
      Think that 1024 is really enough for transfer. Slave has only one executor, so no other jobs on slave are running.

        Attachments

          Issue Links

            Activity

            Hide
            jsirex jsirex added a comment -

            Probably this issue is related to JENKINS-20769

            Show
            jsirex jsirex added a comment - Probably this issue is related to JENKINS-20769
            Hide
            tkopal Tomas Kopal added a comment - - edited

            I do not think its related, at least not directly. We have observed very similar issue. But it exhibits in 1.560 only, after downgrading to 1.559 everything works fine again (no change in plugins). If I understand correctly JENKINS-20769, it was fixed in 1.554, but this really looks like regression in 1.560.

            Apr 23, 2014 4:34:45 PM hudson.slaves.SlaveComputer tryReconnect
            	at hudson.FilePath.act(FilePath.java:893)
            	at hudson.FilePath.act(FilePath.java:909)
            	at hudson.FilePath.archive(FilePath.java:423)
            	at hudson.FilePath.tar(FilePath.java:2047)
            	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:740)
            	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:740)
            	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:710)
            	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:710)
            	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:659)
            	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:659)
            	at hudson.model.Build$BuildExecution.post2(Build.java:182)
            	at hudson.model.Build$BuildExecution.post2(Build.java:182)
            	at hudson.model.Executor.run(Executor.java:231)
            	at hudson.model.Executor.run(Executor.java:231)
            	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
            	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
            	at hudson.model.ResourceController.execute(ResourceController.java:88)
            	at hudson.model.ResourceController.execute(ResourceController.java:88)
            	at hudson.model.Run.execute(Run.java:1728)
            	at hudson.model.Run.execute(Run.java:1728)
            	at hudson.plugins.cloneworkspace.CloneWorkspacePublisher.perform(CloneWorkspacePublisher.java:178)
            	at hudson.plugins.cloneworkspace.CloneWorkspacePublisher.snapshot(CloneWorkspacePublisher.java:231)
            	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
            	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
            	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
            	at hudson.remoting.Channel.call(Channel.java:738)
            	at hudson.remoting.Channel.terminate(Channel.java:801)
            	at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:33)
            	at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:33)
            	at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:33)
            	at hudson.remoting.ChunkedInputStream.read(ChunkedInputStream.java:46)
            	at hudson.remoting.ChunkedInputStream.read(ChunkedInputStream.java:46)
            	at hudson.remoting.ChunkedInputStream.read(ChunkedInputStream.java:46)
            	at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:88)
            	at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:88)
            	at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:88)
            	at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:87)
            	at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:87)
            	at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:87)
            	at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34)
            	at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41)
            	at hudson.remoting.Request.abort(Request.java:299)
            	at hudson.remoting.Request.call(Request.java:174)
            	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
            	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
            	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
            	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:76)
            	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
            	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
            	at hudson.tasks.Mailer.perform(Mailer.java:137)
            	at hudson.tasks.MailSender.createFailureMail(MailSender.java:278)
            	at hudson.tasks.MailSender.execute(MailSender.java:101)
            	at hudson.tasks.MailSender.getMail(MailSender.java:153)
            	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
            	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
            	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
            	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
            	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
            	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
            	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
            	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
            	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
            	at java.util.Arrays.copyOf(Arrays.java:2271)
            	at java.util.Arrays.copyOf(Arrays.java:2271)
            	at java.util.Arrays.copyOf(Arrays.java:2271)
            Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected reader termination
            Caused by: java.io.IOException: Unexpected reader termination
            Caused by: java.lang.OutOfMemoryError: Java heap space
            Exception in thread "Channel reader thread: BM4-Ubuntu_10.04_64" java.lang.OutOfMemoryError: Java heap space
            hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected reader termination
            INFO: Attempting to reconnect BM4-Ubuntu_10.04_64
            java.lang.NullPointerException
            java.lang.OutOfMemoryError: Java heap space
            SEVERE: Unexpected error in channel BM4-Ubuntu_10.04_64
            WARNING: java.io.IOException: Broken pipe
            WARNING: java.io.IOException: Broken pipe
            WARNING: Publisher hudson.plugins.cloneworkspace.CloneWorkspacePublisher aborted due to exception
            WARNING: Publisher hudson.tasks.Mailer aborted due to exception
            
            Show
            tkopal Tomas Kopal added a comment - - edited I do not think its related, at least not directly. We have observed very similar issue. But it exhibits in 1.560 only, after downgrading to 1.559 everything works fine again (no change in plugins). If I understand correctly JENKINS-20769 , it was fixed in 1.554, but this really looks like regression in 1.560. Apr 23, 2014 4:34:45 PM hudson.slaves.SlaveComputer tryReconnect at hudson.FilePath.act(FilePath.java:893) at hudson.FilePath.act(FilePath.java:909) at hudson.FilePath.archive(FilePath.java:423) at hudson.FilePath.tar(FilePath.java:2047) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:740) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:740) at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:710) at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:710) at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:659) at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:659) at hudson.model.Build$BuildExecution.post2(Build.java:182) at hudson.model.Build$BuildExecution.post2(Build.java:182) at hudson.model.Executor.run(Executor.java:231) at hudson.model.Executor.run(Executor.java:231) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Run.execute(Run.java:1728) at hudson.model.Run.execute(Run.java:1728) at hudson.plugins.cloneworkspace.CloneWorkspacePublisher.perform(CloneWorkspacePublisher.java:178) at hudson.plugins.cloneworkspace.CloneWorkspacePublisher.snapshot(CloneWorkspacePublisher.java:231) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.Channel.call(Channel.java:738) at hudson.remoting.Channel.terminate(Channel.java:801) at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:33) at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:33) at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:33) at hudson.remoting.ChunkedInputStream.read(ChunkedInputStream.java:46) at hudson.remoting.ChunkedInputStream.read(ChunkedInputStream.java:46) at hudson.remoting.ChunkedInputStream.read(ChunkedInputStream.java:46) at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:88) at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:88) at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:88) at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:87) at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:87) at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:87) at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34) at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41) at hudson.remoting.Request.abort(Request.java:299) at hudson.remoting.Request.call(Request.java:174) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:76) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.tasks.Mailer.perform(Mailer.java:137) at hudson.tasks.MailSender.createFailureMail(MailSender.java:278) at hudson.tasks.MailSender.execute(MailSender.java:101) at hudson.tasks.MailSender.getMail(MailSender.java:153) at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113) at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113) at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113) at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140) at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140) at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140) at java.util.Arrays.copyOf(Arrays.java:2271) at java.util.Arrays.copyOf(Arrays.java:2271) at java.util.Arrays.copyOf(Arrays.java:2271) Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected reader termination Caused by: java.io.IOException: Unexpected reader termination Caused by: java.lang.OutOfMemoryError: Java heap space Exception in thread "Channel reader thread: BM4-Ubuntu_10.04_64" java.lang.OutOfMemoryError: Java heap space hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected reader termination INFO: Attempting to reconnect BM4-Ubuntu_10.04_64 java.lang.NullPointerException java.lang.OutOfMemoryError: Java heap space SEVERE: Unexpected error in channel BM4-Ubuntu_10.04_64 WARNING: java.io.IOException: Broken pipe WARNING: java.io.IOException: Broken pipe WARNING: Publisher hudson.plugins.cloneworkspace.CloneWorkspacePublisher aborted due to exception WARNING: Publisher hudson.tasks.Mailer aborted due to exception
            Hide
            moa_arosii moa_arosii added a comment -

            We see the same problem when large files are copied to the slave in v. 1.560:

            It is also similar to this recently reported problem: http://jenkins-ci.361315.n4.nabble.com/Problems-with-memory-usage-on-1-560-td4699645.html

            It looks like the problem lies on the slave in this part:

            Caused by: java.lang.OutOfMemoryError: Java heap space
            	at java.util.Arrays.copyOf(Arrays.java:2271)
            	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
            	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
            	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
            	at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:87)}}
            

            If you look on the 'blame' for ChannelBuilder it looks like the usage of FlightRecorderInputStream was introduced in MakeTransport in line 251 20 days ago in commit "Merge branch 'master' into nio" (bdd486dbe6584c607bed0e4324abbd6eeddbb39e).

            It might be a good guess that this "recording' of content causes the problem when large amount of files are transferred. In our case it is close to 100 mb.

            Show
            moa_arosii moa_arosii added a comment - We see the same problem when large files are copied to the slave in v. 1.560: It is also similar to this recently reported problem: http://jenkins-ci.361315.n4.nabble.com/Problems-with-memory-usage-on-1-560-td4699645.html It looks like the problem lies on the slave in this part: Caused by: java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2271) at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113) at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140) at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:87)}} If you look on the 'blame' for ChannelBuilder it looks like the usage of FlightRecorderInputStream was introduced in MakeTransport in line 251 20 days ago in commit "Merge branch 'master' into nio" ( bdd486dbe6584c607bed0e4324abbd6eeddbb39e ). It might be a good guess that this "recording' of content causes the problem when large amount of files are transferred. In our case it is close to 100 mb.
            Hide
            saturnjct Shane Turner added a comment -

            Someone should update the "Affects Version" field to indicate that this is an issue with 1.560.

            Show
            saturnjct Shane Turner added a comment - Someone should update the "Affects Version" field to indicate that this is an issue with 1.560.
            Hide
            peter_vanbiesen Peter Van Biesen added a comment - - edited

            1.559 does not have the problem, 1.560 does. To reproduce, create a job with a shell build step and following code :

            dd if=/dev/urandom of=target.out bs=1M count=50
            for a in 1 2 3 4
            do
            cat target.out >> bigtarget.out
            done
            

            and in post-build actions add a "Archive the artifacts", file : bigtarget.out

            For every build on the same slave, the archived file will increase by 200 Mb.
            Effects :

            • the slave will disconnect
            • the master will not be able to reconnect to the slave UNLESS you choose "configure" on the node, click "save" and then relaunch the slave connection
            • the master will not stop in a normal fashion, only kill -9 will do the trick
            • the slave.jar will keep running on the client and needs to be killed too ( otherwise multiple will keep running )
            Show
            peter_vanbiesen Peter Van Biesen added a comment - - edited 1.559 does not have the problem, 1.560 does. To reproduce, create a job with a shell build step and following code : dd if =/dev/urandom of=target.out bs=1M count=50 for a in 1 2 3 4 do cat target.out >> bigtarget.out done and in post-build actions add a "Archive the artifacts", file : bigtarget.out For every build on the same slave, the archived file will increase by 200 Mb. Effects : the slave will disconnect the master will not be able to reconnect to the slave UNLESS you choose "configure" on the node, click "save" and then relaunch the slave connection the master will not stop in a normal fashion, only kill -9 will do the trick the slave.jar will keep running on the client and needs to be killed too ( otherwise multiple will keep running )
            Hide
            peter_vanbiesen Peter Van Biesen added a comment -

            This issue actually makes jenkins 1.560 unusable I think. Shouldn't be a blocker ?

            Show
            peter_vanbiesen Peter Van Biesen added a comment - This issue actually makes jenkins 1.560 unusable I think. Shouldn't be a blocker ?
            Hide
            lshatzer Larry Shatzer, Jr. added a comment -

            I tried Peter's script with 1.561, and I've run it 3 times. So far run 1 and 2 ran fine. I had to kill run 3, and the slave. Also, the mail-watcher-plugin alerted me to the slave disconnecting and reconnecting... So I've downgraded back to 1.559 again.

            Show
            lshatzer Larry Shatzer, Jr. added a comment - I tried Peter's script with 1.561, and I've run it 3 times. So far run 1 and 2 ran fine. I had to kill run 3, and the slave. Also, the mail-watcher-plugin alerted me to the slave disconnecting and reconnecting... So I've downgraded back to 1.559 again.
            Hide
            efrangul Eric Frangulian added a comment -

            On my side since version 1.560 I am having a memory leak on Jenkins. On the master, looking at the size of the java process, it is 2.6G ( and increasing... 3.3G after 30 minutes ). On 1.559, the size is stable around 1.6G, 1.8G.

            Show
            efrangul Eric Frangulian added a comment - On my side since version 1.560 I am having a memory leak on Jenkins. On the master, looking at the size of the java process, it is 2.6G ( and increasing... 3.3G after 30 minutes ). On 1.559, the size is stable around 1.6G, 1.8G.
            Hide
            corneil_duplessis Corneil du Plessis added a comment -

            Experienced slave disconnecting after builds. Found outofmemory in slave.log.
            Tried 1.561 same problem still persists.
            Rolled back to 1.559

            Show
            corneil_duplessis Corneil du Plessis added a comment - Experienced slave disconnecting after builds. Found outofmemory in slave.log. Tried 1.561 same problem still persists. Rolled back to 1.559
            Show
            jglick Jesse Glick added a comment - https://github.com/jenkinsci/remoting/commit/c0e42ffe12c6bfb3f1e52257589f10dd896817e6 may have fixed this.
            Hide
            jglick Jesse Glick added a comment -

            Remoting 2.38 likely introduced the problem.

            Show
            jglick Jesse Glick added a comment - Remoting 2.38 likely introduced the problem.
            Hide
            gandalf Dell Green added a comment -

            Same problem here, as soon as I upgraded from 1.559 to 1.561 jobs that run daily and have been running for years started failing with out of memory errors on some slaves.

            FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected reader termination
            hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected reader termination
            	at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41)
            	at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34)
            	at hudson.remoting.Request.call(Request.java:174)
            	at hudson.remoting.Channel.call(Channel.java:739)
            	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:168)
            	at com.sun.proxy.$Proxy51.join(Unknown Source)
            	at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:951)
            	at hudson.Launcher$ProcStarter.join(Launcher.java:362)
            	at hudson.tasks.Ant.perform(Ant.java:217)
            	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
            	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:745)
            	at hudson.model.Build$BuildExecution.build(Build.java:198)
            	at hudson.model.Build$BuildExecution.doRun(Build.java:159)
            	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:518)
            	at hudson.model.Run.execute(Run.java:1709)
            	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
            	at hudson.model.ResourceController.execute(ResourceController.java:88)
            	at hudson.model.Executor.run(Executor.java:231)
            Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected reader termination
            	at hudson.remoting.Request.abort(Request.java:299)
            	at hudson.remoting.Channel.terminate(Channel.java:802)
            	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:76)
            Caused by: java.io.IOException: Unexpected reader termination
            	... 1 more
            Caused by: java.lang.OutOfMemoryError: Java heap space
            	at java.util.Arrays.copyOf(Arrays.java:2786)
            	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
            	at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:87)
            	at hudson.remoting.ChunkedInputStream.read(ChunkedInputStream.java:46)
            	at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:88)
            	at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:33)
            	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
            	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
            
            
            
            
            Show
            gandalf Dell Green added a comment - Same problem here, as soon as I upgraded from 1.559 to 1.561 jobs that run daily and have been running for years started failing with out of memory errors on some slaves. FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected reader termination hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected reader termination at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41) at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34) at hudson.remoting.Request.call(Request.java:174) at hudson.remoting.Channel.call(Channel.java:739) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:168) at com.sun.proxy.$Proxy51.join(Unknown Source) at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:951) at hudson.Launcher$ProcStarter.join(Launcher.java:362) at hudson.tasks.Ant.perform(Ant.java:217) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:745) at hudson.model.Build$BuildExecution.build(Build.java:198) at hudson.model.Build$BuildExecution.doRun(Build.java:159) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:518) at hudson.model.Run.execute(Run.java:1709) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:231) Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected reader termination at hudson.remoting.Request.abort(Request.java:299) at hudson.remoting.Channel.terminate(Channel.java:802) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:76) Caused by: java.io.IOException: Unexpected reader termination ... 1 more Caused by: java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2786) at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94) at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:87) at hudson.remoting.ChunkedInputStream.read(ChunkedInputStream.java:46) at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:88) at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:33) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            changelog.html
            pom.xml
            http://jenkins-ci.org/commit/jenkins/aebb876cc82d405022dd3317ece199017d3b7aa8
            Log:
            [FIXED JENKINS-22734] Integrating remoting 2.41 with attempted OOME fix.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: changelog.html pom.xml http://jenkins-ci.org/commit/jenkins/aebb876cc82d405022dd3317ece199017d3b7aa8 Log: [FIXED JENKINS-22734] Integrating remoting 2.41 with attempted OOME fix.
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #3351
            [FIXED JENKINS-22734] Integrating remoting 2.41 with attempted OOME fix. (Revision aebb876cc82d405022dd3317ece199017d3b7aa8)

            Result = SUCCESS
            Jesse Glick : aebb876cc82d405022dd3317ece199017d3b7aa8
            Files :

            • pom.xml
            • changelog.html
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #3351 [FIXED JENKINS-22734] Integrating remoting 2.41 with attempted OOME fix. (Revision aebb876cc82d405022dd3317ece199017d3b7aa8) Result = SUCCESS Jesse Glick : aebb876cc82d405022dd3317ece199017d3b7aa8 Files : pom.xml changelog.html
            Hide
            peter_vanbiesen Peter Van Biesen added a comment -

            1.562 doesn't have this fix yet ...

            Show
            peter_vanbiesen Peter Van Biesen added a comment - 1.562 doesn't have this fix yet ...
            Hide
            cloudyster Marc Cloud added a comment -

            I've installed Jenkins ver. 1.563-SNAPSHOT - works for our installation.

            Got it from

             wget   "https://ci.jenkins-ci.org/view/Jenkins%20core/job/jenkins_rc_branch/lastSuccessfulBuild/artifact/war/target/jenkins.war"
            
            Show
            cloudyster Marc Cloud added a comment - I've installed Jenkins ver. 1.563-SNAPSHOT - works for our installation. Got it from wget "https://ci.jenkins-ci.org/view/Jenkins%20core/job/jenkins_rc_branch/lastSuccessfulBuild/artifact/war/target/jenkins.war"
            Hide
            danielbeck Daniel Beck added a comment -

            This change is scheduled for 1.563 (due to how weird the weekly release process is set up, it takes at least 2-3 releases even for a serious bug to get fixed).

            Show
            danielbeck Daniel Beck added a comment - This change is scheduled for 1.563 (due to how weird the weekly release process is set up, it takes at least 2-3 releases even for a serious bug to get fixed).
            Hide
            myjenkins911 Sridevi Soundarraju added a comment -

            Hi Marc, what other changes did you see after using this SNAPSHOT version? Is there any major change in job configuration? Or something else that I should be aware of before using this version?

            Thanks

            Show
            myjenkins911 Sridevi Soundarraju added a comment - Hi Marc, what other changes did you see after using this SNAPSHOT version? Is there any major change in job configuration? Or something else that I should be aware of before using this version? Thanks
            Hide
            cloudyster Marc Cloud added a comment -

            No abnormalities so far, Sridevi.

            Show
            cloudyster Marc Cloud added a comment - No abnormalities so far, Sridevi.

              People

              • Assignee:
                kohsuke Kohsuke Kawaguchi
                Reporter:
                jsirex jsirex
              • Votes:
                21 Vote for this issue
                Watchers:
                42 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: