Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-1948

Intermittent slave disconnections with secondary symptoms

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Component/s: core
    • Environment:
      Platform: All, OS: All
    • Similar Issues:
      Show 5 results

      Description

      Another intermittent problem.
      Master is linux, target is freebsd 4.9.

      FATAL: Unable to delete script file /var/tmp/hudson60616.sh
      hudson.util.IOException2: remote file operation failed
      at hudson.FilePath.act(FilePath.java:313)
      at hudson.FilePath.delete(FilePath.java:510)
      at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:70)
      at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:34)
      at hudson.model.Build$RunnerImpl.build(Build.java:130)
      at hudson.model.Build$RunnerImpl.doRun(Build.java:105)
      at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:231)
      at hudson.model.Run.run(Run.java:756)
      at hudson.model.Build.run(Build.java:85)
      at hudson.model.ResourceController.execute(ResourceController.java:70)
      at hudson.model.Executor.run(Executor.java:82)
      Caused by: java.io.IOException: already closed
      at hudson.remoting.Channel.send(Channel.java:316)
      at hudson.remoting.Request.call(Request.java:81)
      at hudson.remoting.Channel.call(Channel.java:390)
      at hudson.FilePath.act(FilePath.java:310)
      ... 10 more
      Build was aborted
      FATAL: null
      java.lang.NullPointerException
      at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:78)
      at
      hudson.model.AbstractBuild$AbstractRunner.performAllBuildStep(AbstractBuild.java:309)
      at
      hudson.model.AbstractBuild$AbstractRunner.performAllBuildStep(AbstractBuild.java:297)
      at hudson.model.Build$RunnerImpl.post2(Build.java:118)
      at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:282)
      at hudson.model.Run.run(Run.java:774)
      at hudson.model.Build.run(Build.java:85)
      at hudson.model.ResourceController.execute(ResourceController.java:70)
      at hudson.model.Executor.run(Executor.java:82)

        Attachments

          Issue Links

            Activity

            Hide
            francesco_chemolli Francesco Chemolli added a comment -

            Same issue on OpenBSD 4.8, both 32- and 64-bit.

            Show
            francesco_chemolli Francesco Chemolli added a comment - Same issue on OpenBSD 4.8, both 32- and 64-bit.
            Hide
            vila vila added a comment -

            Same issue here with a master running under Ubuntu natty and slaves (virtualbox VMS) running under FreedBSD, gentoo, ubuntu hardy, lucid, maverick, natty and windows but also with a real host running OSX 10.6.

            I think we can rule out an OS specific issue

            Show
            vila vila added a comment - Same issue here with a master running under Ubuntu natty and slaves (virtualbox VMS) running under FreedBSD, gentoo, ubuntu hardy, lucid, maverick, natty and windows but also with a real host running OSX 10.6. I think we can rule out an OS specific issue
            Hide
            vila vila added a comment -

            So, it seems to be more than intermittent here, anything I can do to help diagnose ?

            It now occurs on a regular basis across all the slaves and may even always for some of them.

            Show
            vila vila added a comment - So, it seems to be more than intermittent here, anything I can do to help diagnose ? It now occurs on a regular basis across all the slaves and may even always for some of them.
            Hide
            vila vila added a comment -

            More evidence on a specific setup:

            One of the slave (macadam.local) is a real host (as opposed to a virtual machine running on the same host than the master).

            It's configure with 'Launch slave via execution of command on the Master' with sh -c "$

            {BABUNE_ROOT}/bin/start-slave-via-ssh.sh macadam.local"

            The start-slave-via-ssh.sh is:
            #!/bin/bash -x

            # This script start a slave via ssh under some assumptions:
            # - slave.jar needs to be up to date,
            # - respect the master ssh config and forward the agent
            NODE=$1

            JAR_REMOTE_PATH='~/babune/slaves/${NODE}/slave.jar'

            # We start in /, let's go to a friendler place
            cd ${BABUNE_ROOT}
            1. Copy the slave.jar file
              scp master/war/WEB-INF/slave.jar $ {NODE}:${JAR_REMOTE_PATH}

              # Start slave.jar
              ssh ${NODE}

              java -jar $

              {JAR_REMOTE_PATH}

            It's needed because I prefer to handle the ssh settings in ~/.ssh/config which is not respected when using 'Launch slave agents on UNIX machines via ssh'

            Now, while this slave is started and a job is running, I started a wireshark on the master with a capture filter of 'host macadam.local and port 22' and also watch the job console displaying progress as tests are passing.

            The suspicious thing there is that while packets are flowing (slowly but flowing) on the ssh connection (as shown by wireshark), no updates happen on the console. Tests generally takes far less than one second and even after waiting for minutes, nothing appears on the console.

            Does that ring some bells ?

            Is there some way to make jenkins a bit more verbose in this area ?

            This seems to indicate that the connection between the master and the slave is broken at a higher level than the ssh one.

            I suspect that the master finally gave up somehow but still try to use the connection to delete the remote file and fail.

            Show
            vila vila added a comment - More evidence on a specific setup: One of the slave (macadam.local) is a real host (as opposed to a virtual machine running on the same host than the master). It's configure with 'Launch slave via execution of command on the Master' with sh -c "$ {BABUNE_ROOT}/bin/start-slave-via-ssh.sh macadam.local" The start-slave-via-ssh.sh is: #!/bin/bash -x # This script start a slave via ssh under some assumptions: # - slave.jar needs to be up to date, # - respect the master ssh config and forward the agent NODE=$1 JAR_REMOTE_PATH='~/babune/slaves/${NODE}/slave.jar' # We start in /, let's go to a friendler place cd ${BABUNE_ROOT} Copy the slave.jar file scp master/war/WEB-INF/slave.jar $ {NODE}:${JAR_REMOTE_PATH} # Start slave.jar ssh ${NODE} java -jar $ {JAR_REMOTE_PATH} It's needed because I prefer to handle the ssh settings in ~/.ssh/config which is not respected when using 'Launch slave agents on UNIX machines via ssh' Now, while this slave is started and a job is running, I started a wireshark on the master with a capture filter of 'host macadam.local and port 22' and also watch the job console displaying progress as tests are passing. The suspicious thing there is that while packets are flowing (slowly but flowing) on the ssh connection (as shown by wireshark), no updates happen on the console. Tests generally takes far less than one second and even after waiting for minutes, nothing appears on the console. Does that ring some bells ? Is there some way to make jenkins a bit more verbose in this area ? This seems to indicate that the connection between the master and the slave is broken at a higher level than the ssh one. I suspect that the master finally gave up somehow but still try to use the connection to delete the remote file and fail.
            Hide
            ustuehler Uwe Stuehler added a comment - - edited

            Some more evidence, yet no solution... :-|

            OpenBSD 5.0-current and Jenkins 1.427.

            In the middle of a kernel build, suddenly there is no activity in
            the console log for a long time and then I get this very similar stack
            trace.

            FATAL: Unable to delete script file /tmp/hudson2998673953817336628.sh
            hudson.util.IOException2: remote file operation failed: /tmp/hudson2998673953817336628.sh at hudson.remoting.Channel@652b32ea:openbsd
            	at hudson.FilePath.act(FilePath.java:754)
            	at hudson.FilePath.act(FilePath.java:740)
            	at hudson.FilePath.delete(FilePath.java:995)
            	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:92)
            	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58)
            	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
            	at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:682)
            	at hudson.model.Build$RunnerImpl.build(Build.java:178)
            	at hudson.model.Build$RunnerImpl.doRun(Build.java:139)
            	at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:448)
            	at hudson.model.Run.run(Run.java:1376)
            	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
            	at hudson.model.ResourceController.execute(ResourceController.java:88)
            	at hudson.model.Executor.run(Executor.java:230)
            Caused by: hudson.remoting.ChannelClosedException: channel is already closed
            	at hudson.remoting.Channel.send(Channel.java:486)
            	at hudson.remoting.Request.call(Request.java:110)
            	at hudson.remoting.Channel.call(Channel.java:668)
            	at hudson.FilePath.act(FilePath.java:747)
            	... 13 more
            Caused by: hudson.remoting.Channel$OrderlyShutdown
            	at hudson.remoting.Channel$CloseCommand.execute(Channel.java:813)
            	at hudson.remoting.Channel$ReaderThread.run(Channel.java:1049)
            Caused by: Command close created at
            [...]
            

            It seems that the build process stalls trying to write to stdout
            because the java slave process no longer consumes the output
            of the build process. Before I the build was aborted, I had a
            chance to look at the process table

            $ pstree
            [...]
             | \-+= 07917 root sshd: hudson [priv] (sshd)
             |   \-+- 26930 hudson sshd: hudson@notty (sshd)
             |     \-+= 03191 hudson ksh -c ksh
             |       \-+- 18833 hudson /home/hudson/jdk/bin/java -jar slave.jar
             |         \-+- 30004 hudson /bin/sh -xe /tmp/hudson2998673953817336628.sh
             |           \-+- 07697 hudson /bin/sh -xe /tmp/hudson2998673953817336628.sh
             |             \--- 16885 hudson make
            [...]
            $ ps auxOwchan|grep make
            hudson   16885  0.0  0.2  2788  3740 ??  I      8:24AM    0:00.04 make             pipewr 
            $ sudo ktrace -p 18833
            $ sudo kdump |head -15
             18833 java     EMUL  "native"
             18833 java     RET   read 51/0x33
             18833 java     CALL  read(0,0x20cd76570,0x2000)
             18833 java     GIO   fd 0 read 696 bytes
                   "xr\0\^Whudson.remoting.Command\0\0\0\0\0\0\0\^A\^B\0\^AL\0      createdAtt\0\^ULjava/lang/Exception;xpsr\0\^^hudson.remoting.Command$Source\0\0\0\0\0\0\0\^A\^B\0\^AL\0\^Fthis$0t\0\^YLhudson/remoting/\
                    Command;xr\0\^Sjava.lang.Exception\M-P\M-}\^_>\^Z;\^\\M-D\^B\0\0xr\0\^Sjava.lang.Throwable\M-U\M-F5'9w\M-8\M-K\^C\0\^CL\0\^Ecauset\0\^ULjava/lang/Throwable;L\0\rdetailMessaget\0\^RLjava/lang/String;[\
                    \0
                    stackTracet\0\^^[Ljava/lang/StackTraceElement;xpq\0~\0\vpur\0\^^[Ljava.lang.StackTraceElement;\^BF*<<\M-}"9\^B\0\0xp\0\0\0\bsr\0\^[java.lang.StackTraceElementa \M-E\M^Z&6\M-]\M^E\^B\0\^DI\0
                    lineNumberL\0\^NdeclaringClassq\0~\0    L\0\bfileNameq\0~\0     L\0
                    methodNameq\0~\0        xp\0\0\0>t\0\^Whudson.remoting.Commandt\0\fCommand.javat\0\^F<init>sq\0~\0\^N\0\0\0/q\0~\0\^Pq\0~\0\^Qq\0~\0\^Rsq\0~\0\^N\0\0\^C)t\0$hudson.remoting.Channel$CloseCommandt\0\fC\
                    hannel.javaq\0~\0\^Rsq\0~\0\^N\0\0\^C)q\0~\0\^Uq\0~\0\^Vq\0~\0\^Rsq\0~\0\^N\0\0\^CV"
             18833 java     RET   read 696/0x2b8
             18833 java     CALL  gettimeofday(0x20cd77c40,0)
             18833 java     PSIG  SIGPROF caught handler=0x2034ee1e0 mask=0x0
             18833 java     RET   gettimeofday 0
            
            Show
            ustuehler Uwe Stuehler added a comment - - edited Some more evidence, yet no solution... :-| OpenBSD 5.0-current and Jenkins 1.427. In the middle of a kernel build, suddenly there is no activity in the console log for a long time and then I get this very similar stack trace. FATAL: Unable to delete script file /tmp/hudson2998673953817336628.sh hudson.util.IOException2: remote file operation failed: /tmp/hudson2998673953817336628.sh at hudson.remoting.Channel@652b32ea:openbsd at hudson.FilePath.act(FilePath.java:754) at hudson.FilePath.act(FilePath.java:740) at hudson.FilePath.delete(FilePath.java:995) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:92) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:682) at hudson.model.Build$RunnerImpl.build(Build.java:178) at hudson.model.Build$RunnerImpl.doRun(Build.java:139) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:448) at hudson.model.Run.run(Run.java:1376) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:230) Caused by: hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:486) at hudson.remoting.Request.call(Request.java:110) at hudson.remoting.Channel.call(Channel.java:668) at hudson.FilePath.act(FilePath.java:747) ... 13 more Caused by: hudson.remoting.Channel$OrderlyShutdown at hudson.remoting.Channel$CloseCommand.execute(Channel.java:813) at hudson.remoting.Channel$ReaderThread.run(Channel.java:1049) Caused by: Command close created at [...] It seems that the build process stalls trying to write to stdout because the java slave process no longer consumes the output of the build process. Before I the build was aborted, I had a chance to look at the process table $ pstree [...] | \-+= 07917 root sshd: hudson [priv] (sshd) | \-+- 26930 hudson sshd: hudson@notty (sshd) | \-+= 03191 hudson ksh -c ksh | \-+- 18833 hudson /home/hudson/jdk/bin/java -jar slave.jar | \-+- 30004 hudson /bin/sh -xe /tmp/hudson2998673953817336628.sh | \-+- 07697 hudson /bin/sh -xe /tmp/hudson2998673953817336628.sh | \--- 16885 hudson make [...] $ ps auxOwchan|grep make hudson 16885 0.0 0.2 2788 3740 ?? I 8:24AM 0:00.04 make pipewr $ sudo ktrace -p 18833 $ sudo kdump |head -15 18833 java EMUL "native" 18833 java RET read 51/0x33 18833 java CALL read(0,0x20cd76570,0x2000) 18833 java GIO fd 0 read 696 bytes "xr\0\^Whudson.remoting.Command\0\0\0\0\0\0\0\^A\^B\0\^AL\0 createdAtt\0\^ULjava/lang/Exception;xpsr\0\^^hudson.remoting.Command$Source\0\0\0\0\0\0\0\^A\^B\0\^AL\0\^Fthis$0t\0\^YLhudson/remoting/\ Command;xr\0\^Sjava.lang.Exception\M-P\M-}\^_>\^Z;\^\\M-D\^B\0\0xr\0\^Sjava.lang.Throwable\M-U\M-F5'9w\M-8\M-K\^C\0\^CL\0\^Ecauset\0\^ULjava/lang/Throwable;L\0\rdetailMessaget\0\^RLjava/lang/String;[\ \0 stackTracet\0\^^[Ljava/lang/StackTraceElement;xpq\0~\0\vpur\0\^^[Ljava.lang.StackTraceElement;\^BF*<<\M-}"9\^B\0\0xp\0\0\0\bsr\0\^[java.lang.StackTraceElementa \M-E\M^Z&6\M-]\M^E\^B\0\^DI\0 lineNumberL\0\^NdeclaringClassq\0~\0 L\0\bfileNameq\0~\0 L\0 methodNameq\0~\0 xp\0\0\0>t\0\^Whudson.remoting.Commandt\0\fCommand.javat\0\^F<init>sq\0~\0\^N\0\0\0/q\0~\0\^Pq\0~\0\^Qq\0~\0\^Rsq\0~\0\^N\0\0\^C)t\0$hudson.remoting.Channel$CloseCommandt\0\fC\ hannel.javaq\0~\0\^Rsq\0~\0\^N\0\0\^C)q\0~\0\^Uq\0~\0\^Vq\0~\0\^Rsq\0~\0\^N\0\0\^CV" 18833 java RET read 696/0x2b8 18833 java CALL gettimeofday(0x20cd77c40,0) 18833 java PSIG SIGPROF caught handler=0x2034ee1e0 mask=0x0 18833 java RET gettimeofday 0
            Hide
            ustuehler Uwe Stuehler added a comment -

            More more food for thought.

            I was accidentally still using pthreads even though I normally replace libpthread
            with librthread during the build. However when the issue came up, it didn't help to
            restart the slave - the issue was showing up consistently in every single build. But
            now that I've actually enabled librthread again and rebooted the slave once more, it
            builds just fine, as usual.

            Show
            ustuehler Uwe Stuehler added a comment - More more food for thought. I was accidentally still using pthreads even though I normally replace libpthread with librthread during the build. However when the issue came up, it didn't help to restart the slave - the issue was showing up consistently in every single build. But now that I've actually enabled librthread again and rebooted the slave once more, it builds just fine, as usual.
            Hide
            gjansen Gerwin Jansen added a comment -

            I have a very similar issue when I use a hudson job to do a very time-consuming svn merge. The console doesn't produce any output when this long svn merge is pending (this is expected). After 38 minutes the below error happens.
            I triggered the job twice, and both failed after 38 minutes.

            System spec
            Jenkins 1.431

            [root@srv-ind-dvm17zl ~]# cat /etc/*release
            Red Hat Enterprise Linux Server release 5.4 (Tikanga)
            [root@srv-ind-dvm17zl ~]# uname -a
            Linux srv-ind-dvm17zl.vanenburg.com 2.6.18-164.el5 #1 SMP Tue Aug 18 15:51:48 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
            [root@srv-ind-dvm17zl ~]# java -version
            java version "1.6.0_20"
            Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
            Java HotSpot(TM) Server VM (build 16.3-b01, mixed mode)
            

            Console log

            setendrevision:
                 [echo]  Prepare DROP-merge command || revision : revision
            [getmergecmd] 
            [getmergecmd]  Merge command: svn merge -r 189757:199394 "http://srv-ind-scrat:8080/repos/bcp/branches/private/batic/batic-wip"  "/opt/jenkins/workspace/batic-wip-dropmerge/trunk" --accept postpone --non-interactive
            
            FATAL: Unable to delete script file /tmp/hudson8787918218652977262.sh
            hudson.util.IOException2: remote file operation failed: /tmp/hudson8787918218652977262.sh at hudson.remoting.Channel@7c1f7480:srv-ind-dvm17zl
            	at hudson.FilePath.act(FilePath.java:754)
            	at hudson.FilePath.act(FilePath.java:740)
            	at hudson.FilePath.delete(FilePath.java:995)
            	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:92)
            	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58)
            	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
            	at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:693)
            	at hudson.model.Build$RunnerImpl.build(Build.java:178)
            	at hudson.model.Build$RunnerImpl.doRun(Build.java:139)
            	at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:459)
            	at hudson.model.Run.run(Run.java:1376)
            	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
            	at hudson.model.ResourceController.execute(ResourceController.java:88)
            	at hudson.model.Executor.run(Executor.java:230)
            Caused by: hudson.remoting.ChannelClosedException: channel is already closed
            	at hudson.remoting.Channel.send(Channel.java:499)
            	at hudson.remoting.Request.call(Request.java:110)
            	at hudson.remoting.Channel.call(Channel.java:681)
            	at hudson.FilePath.act(FilePath.java:747)
            	... 13 more
            Caused by: java.io.IOException: Unexpected termination of the channel
            	at hudson.remoting.Channel$ReaderThread.run(Channel.java:1093)
            Caused by: java.io.EOFException
            	at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2570)
            	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314)
            	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368)
            	at hudson.remoting.Channel$ReaderThread.run(Channel.java:1087)
            FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
            hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
            	at hudson.remoting.Request.call(Request.java:149)
            	at hudson.remoting.Channel.call(Channel.java:681)
            	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:158)
            	at $Proxy38.join(Unknown Source)
            	at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:850)
            	at hudson.Launcher$ProcStarter.join(Launcher.java:336)
            	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:82)
            	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58)
            	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
            	at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:693)
            	at hudson.model.Build$RunnerImpl.build(Build.java:178)
            	at hudson.model.Build$RunnerImpl.doRun(Build.java:139)
            	at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:459)
            	at hudson.model.Run.run(Run.java:1376)
            	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
            	at hudson.model.ResourceController.execute(ResourceController.java:88)
            	at hudson.model.Executor.run(Executor.java:230)
            Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
            	at hudson.remoting.Request.abort(Request.java:273)
            	at hudson.remoting.Channel.terminate(Channel.java:732)
            	at hudson.remoting.Channel$ReaderThread.run(Channel.java:1117)
            Caused by: java.io.IOException: Unexpected termination of the channel
            	at hudson.remoting.Channel$ReaderThread.run(Channel.java:1093)
            Caused by: java.io.EOFException
            	at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2570)
            	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314)
            	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368)
            	at hudson.remoting.Channel$ReaderThread.run(Channel.java:1087)
            
            Show
            gjansen Gerwin Jansen added a comment - I have a very similar issue when I use a hudson job to do a very time-consuming svn merge. The console doesn't produce any output when this long svn merge is pending (this is expected). After 38 minutes the below error happens. I triggered the job twice, and both failed after 38 minutes. System spec Jenkins 1.431 [root@srv-ind-dvm17zl ~]# cat /etc/*release Red Hat Enterprise Linux Server release 5.4 (Tikanga) [root@srv-ind-dvm17zl ~]# uname -a Linux srv-ind-dvm17zl.vanenburg.com 2.6.18-164.el5 #1 SMP Tue Aug 18 15:51:48 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux [root@srv-ind-dvm17zl ~]# java -version java version "1.6.0_20" Java(TM) SE Runtime Environment (build 1.6.0_20-b02) Java HotSpot(TM) Server VM (build 16.3-b01, mixed mode) Console log setendrevision: [echo] Prepare DROP-merge command || revision : revision [getmergecmd] [getmergecmd] Merge command: svn merge -r 189757:199394 "http: //srv-ind-scrat:8080/repos/bcp/branches/ private /batic/batic-wip" "/opt/jenkins/workspace/batic-wip-dropmerge/trunk" --accept postpone --non-interactive FATAL: Unable to delete script file /tmp/hudson8787918218652977262.sh hudson.util.IOException2: remote file operation failed: /tmp/hudson8787918218652977262.sh at hudson.remoting.Channel@7c1f7480:srv-ind-dvm17zl at hudson.FilePath.act(FilePath.java:754) at hudson.FilePath.act(FilePath.java:740) at hudson.FilePath.delete(FilePath.java:995) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:92) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:693) at hudson.model.Build$RunnerImpl.build(Build.java:178) at hudson.model.Build$RunnerImpl.doRun(Build.java:139) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:459) at hudson.model.Run.run(Run.java:1376) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:230) Caused by: hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:499) at hudson.remoting.Request.call(Request.java:110) at hudson.remoting.Channel.call(Channel.java:681) at hudson.FilePath.act(FilePath.java:747) ... 13 more Caused by: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Channel$ReaderThread.run(Channel.java:1093) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2570) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368) at hudson.remoting.Channel$ReaderThread.run(Channel.java:1087) FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Request.call(Request.java:149) at hudson.remoting.Channel.call(Channel.java:681) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:158) at $Proxy38.join(Unknown Source) at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:850) at hudson.Launcher$ProcStarter.join(Launcher.java:336) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:82) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:693) at hudson.model.Build$RunnerImpl.build(Build.java:178) at hudson.model.Build$RunnerImpl.doRun(Build.java:139) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:459) at hudson.model.Run.run(Run.java:1376) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:230) Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Request.abort(Request.java:273) at hudson.remoting.Channel.terminate(Channel.java:732) at hudson.remoting.Channel$ReaderThread.run(Channel.java:1117) Caused by: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Channel$ReaderThread.run(Channel.java:1093) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2570) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368) at hudson.remoting.Channel$ReaderThread.run(Channel.java:1087)
            Hide
            mastkis Kishore Sasidharan added a comment -

            Same issue here with jenkins-1.408 on Centos - slaves on FreeBSD.

            Show
            mastkis Kishore Sasidharan added a comment - Same issue here with jenkins-1.408 on Centos - slaves on FreeBSD.
            Hide
            erik_purins Erik Purins added a comment -

            This is occuring for us on windows slaves with Jenkins ver. 1.420

            Show
            erik_purins Erik Purins added a comment - This is occuring for us on windows slaves with Jenkins ver. 1.420
            Hide
            brianharris brianharris added a comment -

            Suspected duplicate: JENKINS-6817

            Show
            brianharris brianharris added a comment - Suspected duplicate: JENKINS-6817
            Hide
            boinst Ben Ernst added a comment -

            I think I'm seeing much the same issue here, ubuntu master, windows slave, jenkins 1.455.

            09:50:06       [exec] 1>ClCompile:
            09:50:06       [exec] 1>  appsvcs.cpp
            09:50:26       [exec] 1>  All outputs are up-to-date.
            09:52:39  FATAL: Unable to delete script file C:\Windows\TEMP\hudson7112635489663392827.bat
            09:52:39  hudson.util.IOException2: remote file operation failed: C:\Windows\TEMP\hudson7112635489663392827.bat at hudson.remoting.Channel@504f4373:Sue
            09:52:39  	at hudson.FilePath.act(FilePath.java:784)
            09:52:39  	at hudson.FilePath.act(FilePath.java:770)
            09:52:39  	at hudson.FilePath.delete(FilePath.java:1075)
            09:52:39  	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:92)
            09:52:39  	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58)
            09:52:39  	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
            09:52:39  	at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:703)
            09:52:39  	at hudson.model.Build$RunnerImpl.build(Build.java:178)
            09:52:39  	at hudson.model.Build$RunnerImpl.doRun(Build.java:139)
            09:52:39  	at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:473)
            09:52:39  	at hudson.model.Run.run(Run.java:1410)
            09:52:39  	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
            09:52:39  	at hudson.model.ResourceController.execute(ResourceController.java:88)
            09:52:39  	at hudson.model.Executor.run(Executor.java:238)
            09:52:39  Caused by: hudson.remoting.ChannelClosedException: channel is already closed
            09:52:39  	at hudson.remoting.Channel.send(Channel.java:499)
            09:52:39  	at hudson.remoting.Request.call(Request.java:110)
            09:52:39  	at hudson.remoting.Channel.call(Channel.java:681)
            09:52:39  	at hudson.FilePath.act(FilePath.java:777)
            09:52:39  	... 13 more
            09:52:39  Caused by: java.net.SocketException: Connection reset
            09:52:39  	at java.net.SocketInputStream.read(SocketInputStream.java:185)
            09:52:39  	at java.io.FilterInputStream.read(FilterInputStream.java:133)
            09:52:39  	at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
            09:52:39  	at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
            09:52:39  	at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2265)
            09:52:39  	at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2558)
            09:52:39  	at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2568)
            09:52:39  	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314)
            09:52:39  	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368)
            09:52:39  	at hudson.remoting.Channel$ReaderThread.run(Channel.java:1127)
            09:52:39  FATAL: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
            09:52:39  hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
            09:52:39  	at hudson.remoting.Request.call(Request.java:149)
            09:52:39  	at hudson.remoting.Channel.call(Channel.java:681)
            09:52:39  	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:158)
            09:52:39  	at $Proxy42.join(Unknown Source)
            09:52:39  	at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:859)
            09:52:39  	at hudson.Launcher$ProcStarter.join(Launcher.java:345)
            09:52:39  	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:82)
            09:52:39  	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58)
            09:52:39  	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
            09:52:39  	at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:703)
            09:52:39  	at hudson.model.Build$RunnerImpl.build(Build.java:178)
            09:52:39  	at hudson.model.Build$RunnerImpl.doRun(Build.java:139)
            09:52:39  	at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:473)
            09:52:39  	at hudson.model.Run.run(Run.java:1410)
            09:52:39  	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
            09:52:39  	at hudson.model.ResourceController.execute(ResourceController.java:88)
            09:52:39  	at hudson.model.Executor.run(Executor.java:238)
            09:52:39  Caused by: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
            09:52:39  	at hudson.remoting.Request.abort(Request.java:273)
            09:52:39  	at hudson.remoting.Channel.terminate(Channel.java:732)
            09:52:39  	at hudson.remoting.Channel$ReaderThread.run(Channel.java:1157)
            09:52:39  Caused by: java.net.SocketException: Connection reset
            09:52:39  	at java.net.SocketInputStream.read(SocketInputStream.java:185)
            09:52:39  	at java.io.FilterInputStream.read(FilterInputStream.java:133)
            09:52:39  	at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
            09:52:39  	at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
            09:52:39  	at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2265)
            09:52:39  	at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2558)
            09:52:39  	at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2568)
            09:52:39  	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314)
            09:52:39  	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368)
            09:52:39  	at hudson.remoting.Channel$ReaderThread.run(Channel.java:1127)
            
            Show
            boinst Ben Ernst added a comment - I think I'm seeing much the same issue here, ubuntu master, windows slave, jenkins 1.455. 09:50:06 [exec] 1>ClCompile: 09:50:06 [exec] 1> appsvcs.cpp 09:50:26 [exec] 1> All outputs are up-to-date. 09:52:39 FATAL: Unable to delete script file C:\Windows\TEMP\hudson7112635489663392827.bat 09:52:39 hudson.util.IOException2: remote file operation failed: C:\Windows\TEMP\hudson7112635489663392827.bat at hudson.remoting.Channel@504f4373:Sue 09:52:39 at hudson.FilePath.act(FilePath.java:784) 09:52:39 at hudson.FilePath.act(FilePath.java:770) 09:52:39 at hudson.FilePath.delete(FilePath.java:1075) 09:52:39 at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:92) 09:52:39 at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) 09:52:39 at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) 09:52:39 at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:703) 09:52:39 at hudson.model.Build$RunnerImpl.build(Build.java:178) 09:52:39 at hudson.model.Build$RunnerImpl.doRun(Build.java:139) 09:52:39 at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:473) 09:52:39 at hudson.model.Run.run(Run.java:1410) 09:52:39 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) 09:52:39 at hudson.model.ResourceController.execute(ResourceController.java:88) 09:52:39 at hudson.model.Executor.run(Executor.java:238) 09:52:39 Caused by: hudson.remoting.ChannelClosedException: channel is already closed 09:52:39 at hudson.remoting.Channel.send(Channel.java:499) 09:52:39 at hudson.remoting.Request.call(Request.java:110) 09:52:39 at hudson.remoting.Channel.call(Channel.java:681) 09:52:39 at hudson.FilePath.act(FilePath.java:777) 09:52:39 ... 13 more 09:52:39 Caused by: java.net.SocketException: Connection reset 09:52:39 at java.net.SocketInputStream.read(SocketInputStream.java:185) 09:52:39 at java.io.FilterInputStream.read(FilterInputStream.java:133) 09:52:39 at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) 09:52:39 at java.io.BufferedInputStream.read(BufferedInputStream.java:254) 09:52:39 at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2265) 09:52:39 at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2558) 09:52:39 at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2568) 09:52:39 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314) 09:52:39 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368) 09:52:39 at hudson.remoting.Channel$ReaderThread.run(Channel.java:1127) 09:52:39 FATAL: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset 09:52:39 hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset 09:52:39 at hudson.remoting.Request.call(Request.java:149) 09:52:39 at hudson.remoting.Channel.call(Channel.java:681) 09:52:39 at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:158) 09:52:39 at $Proxy42.join(Unknown Source) 09:52:39 at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:859) 09:52:39 at hudson.Launcher$ProcStarter.join(Launcher.java:345) 09:52:39 at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:82) 09:52:39 at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) 09:52:39 at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) 09:52:39 at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:703) 09:52:39 at hudson.model.Build$RunnerImpl.build(Build.java:178) 09:52:39 at hudson.model.Build$RunnerImpl.doRun(Build.java:139) 09:52:39 at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:473) 09:52:39 at hudson.model.Run.run(Run.java:1410) 09:52:39 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) 09:52:39 at hudson.model.ResourceController.execute(ResourceController.java:88) 09:52:39 at hudson.model.Executor.run(Executor.java:238) 09:52:39 Caused by: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset 09:52:39 at hudson.remoting.Request.abort(Request.java:273) 09:52:39 at hudson.remoting.Channel.terminate(Channel.java:732) 09:52:39 at hudson.remoting.Channel$ReaderThread.run(Channel.java:1157) 09:52:39 Caused by: java.net.SocketException: Connection reset 09:52:39 at java.net.SocketInputStream.read(SocketInputStream.java:185) 09:52:39 at java.io.FilterInputStream.read(FilterInputStream.java:133) 09:52:39 at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) 09:52:39 at java.io.BufferedInputStream.read(BufferedInputStream.java:254) 09:52:39 at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2265) 09:52:39 at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2558) 09:52:39 at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2568) 09:52:39 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314) 09:52:39 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368) 09:52:39 at hudson.remoting.Channel$ReaderThread.run(Channel.java:1127)
            Hide
            boinst Ben Ernst added a comment -

            still an issue in jenkins 1.456

            Show
            boinst Ben Ernst added a comment - still an issue in jenkins 1.456
            Hide
            bkrishbnaswamy Balaji Krishnaswamy added a comment -

            Still an issue in jenkins 1.467

            FATAL: Unable to delete script file /var/folders/8j/q0m30zk95rv6_58lnchztz680000gn/T/hudson6575302717651863102.sh
            hudson.util.IOException2: remote file operation failed: /var/folders/8j/q0m30zk95rv6_58lnchztz680000gn/T/hudson6575302717651863102.sh at hudson.remoting.Channel@38526a51:ISTFrameworks-MacMini
            at hudson.FilePath.act(FilePath.java:838)
            at hudson.FilePath.act(FilePath.java:824)
            at hudson.FilePath.delete(FilePath.java:1129)
            at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:92)
            at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58)
            at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
            at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:711)
            at hudson.model.Build$RunnerImpl.build(Build.java:178)
            at hudson.model.Build$RunnerImpl.doRun(Build.java:139)
            at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:481)
            at hudson.model.Run.run(Run.java:1438)
            at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
            at hudson.model.ResourceController.execute(ResourceController.java:88)
            at hudson.model.Executor.run(Executor.java:239)
            Caused by: hudson.remoting.ChannelClosedException: channel is already closed
            at hudson.remoting.Channel.send(Channel.java:475)
            at hudson.remoting.Request.call(Request.java:110)
            at hudson.remoting.Channel.call(Channel.java:646)
            at hudson.FilePath.act(FilePath.java:831)
            ... 13 more
            Caused by: java.net.SocketException: Connection reset
            at java.net.SocketInputStream.read(SocketInputStream.java:189)
            at java.net.SocketInputStream.read(SocketInputStream.java:121)
            at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
            at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
            at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2266)
            at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2559)
            at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2569)
            at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1315)
            at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
            at hudson.remoting.Command.readFrom(Command.java:90)
            at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59)
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
            FATAL: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
            hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
            at hudson.remoting.Request.call(Request.java:149)
            at hudson.remoting.Channel.call(Channel.java:646)
            at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:158)
            at $Proxy141.join(Unknown Source)
            at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:861)
            at hudson.Launcher$ProcStarter.join(Launcher.java:345)
            at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:82)
            at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58)
            at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
            at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:711)
            at hudson.model.Build$RunnerImpl.build(Build.java:178)
            at hudson.model.Build$RunnerImpl.doRun(Build.java:139)
            at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:481)
            at hudson.model.Run.run(Run.java:1438)
            at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
            at hudson.model.ResourceController.execute(ResourceController.java:88)
            at hudson.model.Executor.run(Executor.java:239)
            Caused by: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
            at hudson.remoting.Request.abort(Request.java:273)
            at hudson.remoting.Channel.terminate(Channel.java:702)
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
            Caused by: java.net.SocketException: Connection reset
            at java.net.SocketInputStream.read(SocketInputStream.java:189)
            at java.net.SocketInputStream.read(SocketInputStream.java:121)
            at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
            at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
            at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2266)
            at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2559)
            at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2569)
            at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1315)
            at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
            at hudson.remoting.Command.readFrom(Command.java:90)
            at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59)
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

            Show
            bkrishbnaswamy Balaji Krishnaswamy added a comment - Still an issue in jenkins 1.467 FATAL: Unable to delete script file /var/folders/8j/q0m30zk95rv6_58lnchztz680000gn/T/hudson6575302717651863102.sh hudson.util.IOException2: remote file operation failed: /var/folders/8j/q0m30zk95rv6_58lnchztz680000gn/T/hudson6575302717651863102.sh at hudson.remoting.Channel@38526a51:ISTFrameworks-MacMini at hudson.FilePath.act(FilePath.java:838) at hudson.FilePath.act(FilePath.java:824) at hudson.FilePath.delete(FilePath.java:1129) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:92) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:711) at hudson.model.Build$RunnerImpl.build(Build.java:178) at hudson.model.Build$RunnerImpl.doRun(Build.java:139) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:481) at hudson.model.Run.run(Run.java:1438) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:239) Caused by: hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:475) at hudson.remoting.Request.call(Request.java:110) at hudson.remoting.Channel.call(Channel.java:646) at hudson.FilePath.act(FilePath.java:831) ... 13 more Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:189) at java.net.SocketInputStream.read(SocketInputStream.java:121) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read(BufferedInputStream.java:254) at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2266) at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2559) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2569) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1315) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369) at hudson.remoting.Command.readFrom(Command.java:90) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) FATAL: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset at hudson.remoting.Request.call(Request.java:149) at hudson.remoting.Channel.call(Channel.java:646) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:158) at $Proxy141.join(Unknown Source) at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:861) at hudson.Launcher$ProcStarter.join(Launcher.java:345) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:82) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:711) at hudson.model.Build$RunnerImpl.build(Build.java:178) at hudson.model.Build$RunnerImpl.doRun(Build.java:139) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:481) at hudson.model.Run.run(Run.java:1438) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:239) Caused by: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset at hudson.remoting.Request.abort(Request.java:273) at hudson.remoting.Channel.terminate(Channel.java:702) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69) Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:189) at java.net.SocketInputStream.read(SocketInputStream.java:121) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read(BufferedInputStream.java:254) at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2266) at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2559) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2569) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1315) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369) at hudson.remoting.Command.readFrom(Command.java:90) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
            Hide
            brianfromoregon Brian Harris added a comment -

            For us, the cause of this error was our build slaves (VMs) running out of memory and self-rebooting.

            Show
            brianfromoregon Brian Harris added a comment - For us, the cause of this error was our build slaves (VMs) running out of memory and self-rebooting.
            Hide
            erik_purins Erik Purins added a comment -

            Your VM reboots the operating system? We are not doing that. If the build slave ran out of memory, maybe it is doing something in a buffer it doesn't need to (say, keeping long lines of output it could flush to disk) or has a leak. Since something similar has been happening to us across operating systems, I can help trouble shoot and fix. It's intermittent for us, but it makes our entire build system untrustworthy to our engineers, and they've started ignoring build e-mails.

            Show
            erik_purins Erik Purins added a comment - Your VM reboots the operating system? We are not doing that. If the build slave ran out of memory, maybe it is doing something in a buffer it doesn't need to (say, keeping long lines of output it could flush to disk) or has a leak. Since something similar has been happening to us across operating systems, I can help trouble shoot and fix. It's intermittent for us, but it makes our entire build system untrustworthy to our engineers, and they've started ignoring build e-mails.
            Hide
            sam Soumen Banerjee added a comment -

            I can confirm same issue in AIX and HPUX having failures intermittently. Any pointers for capturing debug information or workaround will be appreciated.

            Show
            sam Soumen Banerjee added a comment - I can confirm same issue in AIX and HPUX having failures intermittently. Any pointers for capturing debug information or workaround will be appreciated.
            Hide
            erik_purins Erik Purins added a comment -

            This is still occurring 1.504 debian squeeze master, osx 10.8 slave. If it's related to an unreliable connection and is this frequent, maybe a better error message (instead of the entire remote action callstack) or more fault-tolerent remote action retry system could be implemented?

            Show
            erik_purins Erik Purins added a comment - This is still occurring 1.504 debian squeeze master, osx 10.8 slave. If it's related to an unreliable connection and is this frequent, maybe a better error message (instead of the entire remote action callstack) or more fault-tolerent remote action retry system could be implemented?
            Hide
            jglick Jesse Glick added a comment -

            JENKINS-5073 massaged the reporting a bit; should no longer give misleading error about deleting script file (which is merely an aftereffect of the slave connectivity issue).

            Show
            jglick Jesse Glick added a comment - JENKINS-5073 massaged the reporting a bit; should no longer give misleading error about deleting script file (which is merely an aftereffect of the slave connectivity issue).
            Hide
            jminne jminne added a comment -

            I'm also seeing this only on an OSX slave. Windows and linux slaves are fine.

            Show
            jminne jminne added a comment - I'm also seeing this only on an OSX slave. Windows and linux slaves are fine.
            Hide
            fdegir Fatih Degirmenci added a comment - - edited

            There are different ways of fixing this issue and in our case, rebooting slaves help. But the other people solved the issue by changing SSH settings on slaves, changing java version used for connecting slaves, etc. (as explained in JENKINS-12235 and other tickets on this issue.)

            So it is crucial for everyone that this issue is solved.

            Of course there are some things which Jenkins can not solve such as issues with the slaves themselves, etc. If this is the case, it would be good if Jenkins tells us a bit more. The error message printed to console could tell us what the "real" problem is, rather than just saying unable to delete the script file. This misleads us and we start troubleshooting unrelated things rather than just checking the slave and rebooting it if necessary. So, having an indication regarding what's the problem behind this preventing Jenkins from deleting the file could improve things. (is it connectivity issue, etc and so on.)

            Show
            fdegir Fatih Degirmenci added a comment - - edited There are different ways of fixing this issue and in our case, rebooting slaves help. But the other people solved the issue by changing SSH settings on slaves, changing java version used for connecting slaves, etc. (as explained in JENKINS-12235 and other tickets on this issue.) So it is crucial for everyone that this issue is solved. Of course there are some things which Jenkins can not solve such as issues with the slaves themselves, etc. If this is the case, it would be good if Jenkins tells us a bit more. The error message printed to console could tell us what the "real" problem is, rather than just saying unable to delete the script file. This misleads us and we start troubleshooting unrelated things rather than just checking the slave and rebooting it if necessary. So, having an indication regarding what's the problem behind this preventing Jenkins from deleting the file could improve things. (is it connectivity issue, etc and so on.)
            Hide
            guyr Guy Rozendorn added a comment - - edited

            This week we changed all our 80± slaves from using the SSHLauncher to use the CommandLauncher, which launches strace -t -s 4096 ssh ..., which the following lines in .ssh/config:

            TCPKeepAlive yes
            ServerAliveInterval 10
            ServerAliveCountMax 10
            LogLevel DEBUG
            

            The reason using strace is to get a clue if the connection is dropped first, or the master decides it is dead.

            In one of the job executions (which started at 00:00:56 we get this in the log:

            Started by timer
            Building remotely on host-ci66 in workspace /root/jenkins/workspace/mainline-bdist-develop
            
            Deleting project workspace... Checkout:mainline-bdist-develop / /root/jenkins/workspace/mainline-bdist-develop - hudson.remoting.Channel@27ecbe67:host-ci66
            Using strategy: Default
            Last Built Revision: Revision ff29df8b003dde47573ddfb0b463351baee6dea3 (origin/develop)
            FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
            hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
            	at hudson.remoting.Request.call(Request.java:174)
            	at hudson.remoting.Channel.call(Channel.java:722)
            	at hudson.FilePath.act(FilePath.java:894)
            	at hudson.FilePath.act(FilePath.java:878)
            	at hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:942)
            	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1108)
            	at hudson.model.AbstractProject.checkout(AbstractProject.java:1369)
            	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:676)
            	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
            	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:581)
            	at hudson.model.Run.execute(Run.java:1593)
            	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
            	at hudson.model.ResourceController.execute(ResourceController.java:88)
            	at hudson.model.Executor.run(Executor.java:242)
            Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
            	at hudson.remoting.Request.abort(Request.java:299)
            	at hudson.remoting.Channel.terminate(Channel.java:782)
            	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
            Caused by: java.io.IOException: Unexpected termination of the channel
            	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
            Caused by: java.io.EOFException
            	at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2595)
            	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1315)
            	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
            	at hudson.remoting.Command.readFrom(Command.java:92)
            	at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59)
            	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
            

            and the slave.log ends with this:

            00:00:56 select(7, [3 4], [3 5], NULL, {10, 0}) = 4 (in [3 4], out [3 5], left {9, 999997})
            00:00:56 read(4, "\10\0\245\231\255B,\304\304\7\23\3\0\0000\7\0\0d\0\0\0hudson/plugins/git/browser/FisheyeGitRepositoryBrowser$FisheyeGitRepositoryBrowserDescriptor$1.class\265U{O\23A\20\377m)\\\251\247\205\"\240@\364\264UK\21\17|\240\370\226\332\"Z5\341\225h\214\361\350-\355\312q[\357\256\5\376\364c\370-0\0214\232\370\1\374P\306\331k!&\0264&m\2567\2733\263\363\370\315\314\336\217\237_\277\3\230\304\223n\34\303\250ze\343H`L\303\3058\242\30\217\23\347R\f\246\242\23qR\274\34\303\25E\257j\270\246\230S\32\256k\270\301p\254n9\351U\341Z\316\262\345\3248C\262\370\326\252[\246c\271es!\360\204[\276\305\320\25T\204\237\236\320@\353s\5\341W\370\26\237\25\301<\257J_\4\322\333\232\361\344\206\317\275\207\334/y\242J\34\6}\316u\271\227s,\337\347>\303\353b\245f\373\3225\253N\255,\\\337,\213\300\\i\0343\0171\231\376'w*\306\333\302\25\301]\6?\323^W\177\0024\272\314\20\315I\233\340K\24\205\313\237\325\326W\270\267h\2558!\240\262\244\320\365\204\3327\231Q\5(\3\30\336\2645\330\364$!\323Y\252\360\322\32\303\251\314\350\236\263Z \34\263 \275u\252\272\260\255@H\227\24\31E\326\337HOHs\356y~\263\304\253MY|\177\343k\270M\373\5Y\363J\274 T:\306!\301\\R\366\10\230\274[rH\342\226\237\362\240\"m\35wpWG\22\3:z\320\253c\20\367\250iB\347.\17\314\245\371\242\342\335\327\361\0003\f\232\362\220\337\342:rx\250!\257\243\0\203\341\352\"\341h\320c\31u\225\211A\307\214\225Z`\210\300\260%\367\335\v\201\341H\271f8b\215\33\3736fA}\322\327\"Q\35\2170G\315\332\336\2320\244\16\256C\232R\3105\n\26\333[jx\314\360\252\235AQw(86M\342\326\35\302\177\241A\367\221\241\212\3332\ff\311s\30&2-\306\340\360\356\352\310\2509\351ouP\315\217\254rr\222i\212\233-@6\367\2524S[]\345\36\267\347\271es5\3601\272\266\354E\276\0310\214g\16Rk\341\354%\303\320\301\2012D$A\337\311=O\241\322[\261\\\333\341\277\265\10C\276E\n-\247\346/\200L\375_Aa\320\5\236\240\373?\322\323\243F\7 Js\2448\364\37\304\t\272VN\322j\232\366\35D\23\331\261\35\260\354\305\35D\262\237\321\3611T\34\242w\27)\2\353\30\246\267\36\256\23\30\301)\242Q\234&7\221\320\314{t\206\206\307\263\331o\210\276\310~B\3443:w\321\225\324v\21\373\0\355\v\272\267\223\361/8\262]\314*\351\330.\216n\207G\206q\226\f\217\340LH\33N\7\310\34\360\216\242\364H\22\220\244\216\0246\302 \6H'\205x\250\257\302\31o\206\223\n\203\215\220\2154et.L\340<.\204\201\366!\203~Z\r\221\244\0177q\34\352{\330\370%\350\2237\35\355\376\5PK\3\4\n\0\0\10\10\0\245\231\255B\342o\20\6\230\5\0\0\314\f\0\0b\0\0\0hudson/plugins/git/browser/FisheyeGitRepositoryBrowser$FisheyeGitRepositoryBrowserDescriptor.class\265W{S\23W\24\377\255\0016\204E\20\37(\365\21-\266!\321\254\257\252m(-\362\20h\2H\20|\353&\271$\0276\273q\37H\332\332\367\373\365\1\374\24jg\"\326\231N\377\353L?T\247\347nB\0101\240\343\324\314\354}\236\347\357\236s\356\315?\377\376\361'\200\223\370-\200\3\30\363c\\\306\204\37\37\265!\216D\0\223\230\362c:\200K\230\221\221\f\240\25c\242\231\25\315e\321\314\265b?\346\3\270\202\253\1\\\303u?n\10\256\233\1\334\302\355Vj\356\210\221&#\25\300\36dD\303\2\304\262\340GVp\345dp\31\213\22\332.\317\304oO\17\316\316\216\314LJ\330\27_\324\2265\325u\270\256Z,\313V\324i\315q\230e\304$\264\364s\203;\3\22|\241\2769\tMCf\206I\350\210s\203M\272\371\24\263f\265\224N+]q3\255\351s\232\305\305\274\262\330\344\344\270-\341\350(\267s\254\310.rg\206\25L\233;\246U\274`\231\367lf\r3;m\361\2\255HP\306\r\203YC\272f\333\214\330n\305sn\3066\r\265\240\273Yn\330j\226;j\252\314\246n!\262\367\245\324\221o\333\263\314\31\346vA\327\212\223Z\236\354\335\25\352+C\241kFVM:\0267\262D\327f\260{\343\206\355hF\232\210\226Cq\323\312\252Kf\316v\227\230J\313\5\235\fJ\226\373\31v\327e\266\23\213\33\314Q\355\5uQ80\221\234\232\234J-\262\264\23\353{E\247\310\f\237\305\356J\350}\31\355\22\2BqY\247\204\275\233YCt#+iVp\270i\3302\226h\2361\207r,\275t\331\322%\234\10=\17G\325\1/\\FM+?\247\351<\243\t\21$\257yY\323]/\36\32\0\31X\340\206\10\22\242\220\221\227aH82\343\32\16\317\2639ns\212\231i\315\242\223\240\330\0334\f\323\361\204\332\233\371|\311eV\261\312@\342\375\v|e$_p\212>J2I\302\225\377\347\244\362\24\362\272Z\216\234\224\10lR\265m\345\204hN\222\322\376\264^I\221\326$\317\32\232\343Z\344\177\177#n\21w\325\r;\235W\237;\346\376\360@l@ \2254]+\3ERROR: Connection terminated
            ESC[8mha:AAAAWB+LCAAAAAAAAABb85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0ldFVf2c+b/lb5MDAwVRQxSaBqcITRIIQMEMIIUFgAAckCEiWAAAAA=ESC[0mjava.io.IOException: Unexpected termination of the channel
                    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
            Caused by: java.io.EOFException
                    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2595)
                    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1315)
                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
                    at hudson.remoting.Command.readFrom(Command.java:92)
                    at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59)
                    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
            

            From what I understand, when the java process on the master node decides that the connection is terminated, the SSH connection was still up and running and with data flowing in both directions.

            Would it help if we dump the stdout and stdin streams into files and attach them to this ticket?

            Show
            guyr Guy Rozendorn added a comment - - edited This week we changed all our 80± slaves from using the SSHLauncher to use the CommandLauncher, which launches strace -t -s 4096 ssh ... , which the following lines in .ssh/config : TCPKeepAlive yes ServerAliveInterval 10 ServerAliveCountMax 10 LogLevel DEBUG The reason using strace is to get a clue if the connection is dropped first, or the master decides it is dead. In one of the job executions (which started at 00:00:56 we get this in the log: Started by timer Building remotely on host-ci66 in workspace /root/jenkins/workspace/mainline-bdist-develop Deleting project workspace... Checkout:mainline-bdist-develop / /root/jenkins/workspace/mainline-bdist-develop - hudson.remoting.Channel@27ecbe67:host-ci66 Using strategy: Default Last Built Revision: Revision ff29df8b003dde47573ddfb0b463351baee6dea3 (origin/develop) FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Request.call(Request.java:174) at hudson.remoting.Channel.call(Channel.java:722) at hudson.FilePath.act(FilePath.java:894) at hudson.FilePath.act(FilePath.java:878) at hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:942) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1108) at hudson.model.AbstractProject.checkout(AbstractProject.java:1369) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:676) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:581) at hudson.model.Run.execute(Run.java:1593) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:242) Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Request.abort(Request.java:299) at hudson.remoting.Channel.terminate(Channel.java:782) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69) Caused by: java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2595) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1315) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) and the slave.log ends with this: 00:00:56 select(7, [3 4], [3 5], NULL, {10, 0}) = 4 (in [3 4], out [3 5], left {9, 999997}) 00:00:56 read(4, "\10\0\245\231\255B,\304\304\7\23\3\0\0000\7\0\0d\0\0\0hudson/plugins/git/browser/FisheyeGitRepositoryBrowser$FisheyeGitRepositoryBrowserDescriptor$1.class\265U{O\23A\20\377m)\\\251\247\205\"\240@\364\264UK\21\17|\240\370\226\332\"Z5\341\225h\214\361\350-\355\312q[\357\256\5\376\364c\370-0\0214\232\370\1\374P\306\331k!&\0264&m\2567\2733\263\363\370\315\314\336\217\237_\277\3\230\304\223n\34\303\250ze\343H`L\303\3058\242\30\217\23\347R\f\246\242\23qR\274\34\303\25E\257j\270\246\230S\32\256k\270\301p\254n9\351U\341Z\316\262\345\3248C\262\370\326\252[\246c\271es!\360\204[\276\305\320\25T\204\237\236\320@\353s\5\341W\370\26\237\25\301<\257J_\4\322\333\232\361\344\206\317\275\207\334/y\242J\34\6}\316u\271\227s,\337\347>\303\353b\245f\373\3225\253N\255,\\\337,\213\300\\i\0343\0171\231\376'w*\306\333\302\25\301]\6?\323^W\177\0024\272\314\20\315I\233\340K\24\205\313\237\325\326W\270\267h\2558!\240\262\244\320\365\204\3327\231Q\5(\3\30\336\2645\330\364$!\323Y\252\360\322\32\303\251\314\350\236\263Z \34\263 \275u\252\272\260\255@H\227\24\31E\326\337HOHs\356y~\263\304\253MY|\177\343k\270M\373\5Y\363J\274 T:\306!\301\\R\366\10\230\274[rH\342\226\237\362\240\"m\35wpWG\22\3:z\320\253c\20\367\250iB\347.\17\314\245\371\242\342\335\327\361\0003\f\232\362\220\337\342:rx\250!\257\243\0\203\341\352\"\341h\320c\31u\225\211A\307\214\225Z`\210\300\260%\367\335\v\201\341H\271f8b\215\33\3736fA}\322\327\"Q\35\2170G\315\332\336\2320\244\16\256C\232R\3105\n\26\333[jx\314\360\252\235AQw(86M\342\326\35\302\177\241A\367\221\241\212\3332\ff\311s\30&2-\306\340\360\356\352\310\2509\351ouP\315\217\254rr\222i\212\233-@6\367\2524S[]\345\36\267\347\271es5\3601\272\266\354E\276\0310\214g\16Rk\341\354%\303\320\301\2012D$A\337\311=O\241\322[\261\\\333\341\277\265\10C\276E\n-\247\346/\200L\375_Aa\320\5\236\240\373?\322\323\243F\7 Js\2448\364\37\304\t\272VN\322j\232\366\35D\23\331\261\35\260\354\305\35D\262\237\321\3611T\34\242w\27)\2\353\30\246\267\36\256\23\30\301)\242Q\234&7\221\320\314{t\206\206\307\263\331o\210\276\310~B\3443:w\321\225\324v\21\373\0\355\v\272\267\223\361/8\262]\314*\351\330.\216n\207G\206q\226\f\217\340LH\33N\7\310\34\360\216\242\364H\22\220\244\216\0246\302 \6H'\205x\250\257\302\31o\206\223\n\203\215\220\2154et.L\340<.\204\201\366!\203~Z\r\221\244\0177q\34\352{\330\370%\350\2237\35\355\376\5PK\3\4\n\0\0\10\10\0\245\231\255B\342o\20\6\230\5\0\0\314\f\0\0b\0\0\0hudson/plugins/git/browser/FisheyeGitRepositoryBrowser$FisheyeGitRepositoryBrowserDescriptor.class\265W{S\23W\24\377\255\0016\204E\20\37(\365\21-\266!\321\254\257\252m(-\362\20h\2H\20|\353&\271$\0276\273q\37H\332\332\367\373\365\1\374\24jg\"\326\231N\377\353L?T\247\347nB\0101\240\343\324\314\354}\236\347\357\236s\356\315?\377\376\361'\200\223\370-\200\3\30\363c\\\306\204\37\37\265!\216D\0\223\230\362c:\200K\230\221\221\f\240\25c\242\231\25\315e\321\314\265b?\346\3\270\202\253\1\\\303u?n\10\256\233\1\334\302\355Vj\356\210\221&#\25\300\36dD\303\2\304\262\340GVp\345dp\31\213\22\332.\317\304oO\17\316\316\216\314LJ\330\27_\324\2265\325u\270\256Z,\313V\324i\315q\230e\304$\264\364s\203;\3\22|\241\2769\tMCf\206I\350\210s\203M\272\371\24\263f\265\224N+]q3\255\351s\232\305\305\274\262\330\344\344\270-\341\350(\267s\254\310.rg\206\25L\233;\246U\274`\231\367lf\r3;m\361\2\255HP\306\r\203YC\272f\333\214\330n\305sn\3066\r\265\240\273Yn\330j\226;j\252\314\246n!\262\367\245\324\221o\333\263\314\31\346vA\327\212\223Z\236\354\335\25\352+C\241kFVM:\0267\262D\327f\260{\343\206\355hF\232\210\226Cq\323\312\252Kf\316v\227\230J\313\5\235\fJ\226\373\31v\327e\266\23\213\33\314Q\355\5uQ80\221\234\232\234J-\262\264\23\353{E\247\310\f\237\305\356J\350}\31\355\22\2BqY\247\204\275\233YCt#+iVp\270i\3302\226h\2361\207r,\275t\331\322%\234\10=\17G\325\1/\\FM+?\247\351<\243\t\21$\257yY\323]/\36\32\0\31X\340\206\10\22\242\220\221\227aH82\343\32\16\317\2639ns\212\231i\315\242\223\240\330\0334\f\323\361\204\332\233\371|\311eV\261\312@\342\375\v|e$_p\212>J2I\302\225\377\347\244\362\24\362\272Z\216\234\224\10lR\265m\345\204hN\222\322\376\264^I\221\326$\317\32\232\343Z\344\177\177#n\21w\325\r;\235W\237;\346\376\360@l@ \2254]+\3ERROR: Connection terminated ESC[8mha:AAAAWB+LCAAAAAAAAABb85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0ldFVf2c+b/lb5MDAwVRQxSaBqcITRIIQMEMIIUFgAAckCEiWAAAAA=ESC[0mjava.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2595) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1315) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) From what I understand, when the java process on the master node decides that the connection is terminated, the SSH connection was still up and running and with data flowing in both directions. Would it help if we dump the stdout and stdin streams into files and attach them to this ticket?

              People

              • Assignee:
                oleg_nenashev Oleg Nenashev
                Reporter:
                bll6969 bll
              • Votes:
                35 Vote for this issue
                Watchers:
                32 Start watching this issue

                Dates

                • Created:
                  Updated: