Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-71133

Slave disconnect unexpectedly hudson.remoting.ChannelClosedException

XMLWordPrintable

      Hi,

      Every couple of days my jenkins slave got disconnected while running builds.

      On the slave build log I have this:

      02:55:25 FATAL: command execution failed
02:55:25 java.io.EOFException
02:55:25 	at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2832)
02:55:25 	at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3307)
02:55:25 	at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:934)
02:55:25 	at java.io.ObjectInputStream.<init>(ObjectInputStream.java:396)
02:55:25 	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
02:55:25 	at hudson.remoting.Command.readFrom(Command.java:142)
02:55:25 	at hudson.remoting.Command.readFrom(Command.java:128)
02:55:25 	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
02:55:25 	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
02:55:25 Caused: java.io.IOException: Unexpected termination of the channel
02:55:25 	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
02:55:25 Caused: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@565f8d19:QA Automation": Remote call on QA Automation failed. The channel is closing down or has closed down
02:55:25 	at hudson.remoting.Channel.call(Channel.java:994)
02:55:25 	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286)
02:55:25 	at com.sun.proxy.$Proxy106.isAlive(Unknown Source)
02:55:25 	at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1211)
02:55:25 	at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1203)
02:55:25 	at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:194)
02:55:25 	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:144)
02:55:25 	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:91)
02:55:25 	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
02:55:25 	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741)
02:55:25 	at hudson.model.Build$BuildExecution.build(Build.java:206)
02:55:25 	at hudson.model.Build$BuildExecution.doRun(Build.java:163)
02:55:25 	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
02:55:25 	at hudson.model.Run.execute(Run.java:1894)
02:55:25 	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
02:55:25 	at hudson.model.ResourceController.execute(ResourceController.java:97)
02:55:25 	at hudson.model.Executor.run(Executor.java:428)
02:55:25 FATAL: Unable to delete script file /tmp/jenkins2698528342924960038.sh
02:55:25 java.io.EOFException
02:55:25 	at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2832)
02:55:25 	at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3307)
02:55:25 	at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:934)
02:55:25 	at java.io.ObjectInputStream.<init>(ObjectInputStream.java:396)
02:55:25 	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
02:55:25 	at hudson.remoting.Command.readFrom(Command.java:142)
02:55:25 	at hudson.remoting.Command.readFrom(Command.java:128)
02:55:25 	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
02:55:25 	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
02:55:25 Caused: java.io.IOException: Unexpected termination of the channel
02:55:25 	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
02:55:25 Caused: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@565f8d19:QA Automation": Remote call on QA Automation failed. The channel is closing down or has closed down
02:55:25 	at hudson.remoting.Channel.call(Channel.java:994)
02:55:25 	at hudson.FilePath.act(FilePath.java:1070)
02:55:25 	at hudson.FilePath.act(FilePath.java:1059)
02:55:25 	at hudson.FilePath.delete(FilePath.java:1582)
02:55:25 	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:162)
02:55:25 	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:91)
02:55:25 	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
02:55:25 	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741)
02:55:25 	at hudson.model.Build$BuildExecution.build(Build.java:206)
02:55:25 	at hudson.model.Build$BuildExecution.doRun(Build.java:163)
02:55:25 	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
02:55:25 	at hudson.model.Run.execute(Run.java:1894)
02:55:25 	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
02:55:25 	at hudson.model.ResourceController.execute(ResourceController.java:97)
02:55:25 	at hudson.model.Executor.run(Executor.java:428)
02:55:25 Build step 'Execute shell' marked build as failure
02:55:25 Notifying upstream projects of job completion
02:55:25 Finished: FAILURE

      On the server log I got this:

      2023-04-23 23:55:25.058+0000 [id=2242399]    INFO    h.r.SynchronousCommandTransport$ReaderThread#run: I/O error in channel QA Automation
java.io.EOFException
    at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2832)
    at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3307)
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:934)
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:396)
    at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
    at hudson.remoting.Command.readFrom(Command.java:142)
    at hudson.remoting.Command.readFrom(Command.java:128)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Caused: java.io.IOException: Unexpected termination of the channel
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
2023-04-23 23:55:57.079+0000 [id=38]    INFO    hudson.slaves.SlaveComputer#tryReconnect: Attempting to reconnect QA Automation
[04/24/23 02:56:39] SSH Launch of QA Automation on 54.176.197.167 completed in 42,886 ms

      Jenkins server version: 2.249.2

      SSH Agent version: 1.20

      SSH plugin version: 2.6.1

       

      I some some other tickets about this issue but with no clear solution, like:

      https://issues.jenkins.io/browse/JENKINS-25858

      https://issues.jenkins.io/browse/JENKINS-48850

       

      My builds are actully running docker, when the dissconnection happens, the builds stopped but the containers keep running.

      Can you please assist?

            jthompson Jeff Thompson
            avie Avie
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: