Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-30869

After upgrade from 1.574 to 1.609.3, cannot connect to Linux slave

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Component/s: remoting
    • Environment:
      Host: Ubuntu 12.04 x64 Server
      Slave: Linux freescale 2.6.31 armv5tejl
    • Similar Issues:

      Description

      Sometimes I see this error:

      [10/09/15 14:38:27] [SSH] Opening SSH connection to 10.6.5.9:22.
      [10/09/15 14:38:28] [SSH] Authentication successful.
      [10/09/15 14:38:28] [SSH] The remote users environment is:
      HOME='/home/myaccount'
      IFS='
      '
      LOGNAME='myaccount'
      OPTIND='1'
      PATH='/usr/bin:/bin'
      PPID='9230'
      PS1='\w \$ '
      PS2='> '
      PS4='+ '
      PWD='/home/myaccount'
      SHELL='/bin/sh'
      USER='myaccount'
      [10/09/15 14:38:28] [SSH] Starting sftp client.
      [10/09/15 14:38:28] [SSH] SFTP failed. Copying via SCP.
      [10/09/15 14:38:28] [SSH] Copying latest slave.jar...
      Expanded the channel window size to 4MB
      [10/09/15 14:38:28] [SSH] Starting slave process: cd "/home/myaccount" && /opt/jre/bin/java -jar slave.jar
      <===[JENKINS REMOTING CAPACITY]===>channel started
      Slave.jar version: 2.52
      This is a Unix slave
      Segmentation fault
      ERROR: Connection terminated
      java.io.IOException: Unexpected EOF
      at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:99)
      at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
      at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
      at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
      ERROR: Failed to install restarter
      hudson.remoting.ChannelClosedException: channel is already closed
      at hudson.remoting.Channel.send(Channel.java:575)
      at hudson.remoting.Request.call(Request.java:130)
      at hudson.remoting.Channel.call(Channel.java:777)
      at jenkins.slaves.restarter.JnlpSlaveRestarterInstaller.install(JnlpSlaveRestarterInstaller.java:52)
      at jenkins.slaves.restarter.JnlpSlaveRestarterInstaller.access$000(JnlpSlaveRestarterInstaller.java:33)
      at jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$1.call(JnlpSlaveRestarterInstaller.java:39)
      at jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$1.call(JnlpSlaveRestarterInstaller.java:36)
      at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:744)
      Caused by: java.io.IOException: Unexpected EOF
      at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:99)
      at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
      at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
      at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
      Slave successfully connected and online
      ERROR: Failed to update jenkins-slave.exe
      Slave JVM has terminated. Exit code=139
      java.io.IOException: remote file operation failed: /home/myaccount/jenkins-slave.exe at hudson.remoting.Channel@3902ea91:Freescale-5.9: hudson.remoting.ChannelClosedException: channel is already closed
      at hudson.FilePath.act(FilePath.java:985)
      at hudson.FilePath.act(FilePath.java:967)
      at hudson.FilePath.exists(FilePath.java:1435)
      at org.jenkinsci.modules.windows_slave_installer.SlaveExeUpdater$1.call(SlaveExeUpdater.java:46)
      at org.jenkinsci.modules.windows_slave_installer.SlaveExeUpdater$1.call(SlaveExeUpdater.java:37)
      at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:744)
      Caused by: hudson.remoting.ChannelClosedException: channel is already closed
      at hudson.remoting.Channel.send(Channel.java:575)
      at hudson.remoting.Request.call(Request.java:130)
      at hudson.remoting.Channel.call(Channel.java:777)
      at hudson.FilePath.act(FilePath.java:978)
      ... 9 more
      Caused by: java.io.IOException: Unexpected EOF
      at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:99)
      at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
      at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
      at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
      [10/09/15 14:39:03] [SSH] Connection closed.

      Sometimes I see this, though:

      [10/09/15 14:45:28] [SSH] Starting sftp client.
      [10/09/15 14:45:28] [SSH] SFTP failed. Copying via SCP.
      [10/09/15 14:45:28] [SSH] Copying latest slave.jar...
      Expanded the channel window size to 4MB
      [10/09/15 14:45:28] [SSH] Starting slave process: cd "/home/myaccount" && /opt/jre/bin/java -jar slave.jar
      <===[JENKINS REMOTING CAPACITY]===>channel started
      Slave.jar version: 2.52
      This is a Unix slave
      Segmentation fault
      ERROR: Connection terminated
      java.io.IOException: Unexpected EOF
      at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:99)
      at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
      at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
      at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
      ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins.
      java.lang.NullPointerException
      at org.jenkinsci.modules.systemd_slave_installer.SlaveInstallerFactoryImpl.createIfApplicable(SlaveInstallerFactoryImpl.java:33)
      at org.jenkinsci.modules.slave_installer.SlaveInstallerFactory.createIfApplicable(SlaveInstallerFactory.java:29)
      at org.jenkinsci.modules.slave_installer.SlaveInstallerFactory.createFor(SlaveInstallerFactory.java:46)
      at org.jenkinsci.modules.slave_installer.impl.ComputerListenerImpl.onOnline(ComputerListenerImpl.java:30)
      at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:563)
      at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:371)
      at hudson.plugins.sshslaves.SSHLauncher.startSlave(SSHLauncher.java:945)
      at hudson.plugins.sshslaves.SSHLauncher.access$400(SSHLauncher.java:133)
      at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:711)
      at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:696)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:744)
      [10/09/15 14:46:04] Launch failed - cleaning up connection
      [10/09/15 14:46:04] [SSH] Connection closed.

      Before I updated to 1.609.3, it worked without issue.

      Java is: java version 1.7.0_60
      Java SE Embedded Runtime Environment (build 1.7.0_60-b19, headless)

        Attachments

          Activity

          Hide
          jsposato John Sposato added a comment -

          I am seeing similar behavior on a CentOS 7 slave from Jenkins 1.633. Job errors during archiving with this:

          ERROR: Failed to archive artifacts: android/app/build/outputs/apk/*.apk
          java.io.IOException: java.io.IOException: Failed to extract /var/lib/jenkins/workspace/Wawa_Android_QA/transfer of 1 files
          	at hudson.FilePath.readFromTar(FilePath.java:2301)
          	at hudson.FilePath.copyRecursiveTo(FilePath.java:2210)
          	at jenkins.model.StandardArtifactManager.archive(StandardArtifactManager.java:61)
          	at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:219)
          	at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:75)
          	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
          	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
          	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:726)
          	at hudson.model.Build$BuildExecution.post2(Build.java:185)
          	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:671)
          	at hudson.model.Run.execute(Run.java:1766)
          	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          	at hudson.model.ResourceController.execute(ResourceController.java:98)
          	at hudson.model.Executor.run(Executor.java:408)
          Caused by: java.io.IOException: java.io.IOException: Unexpected EOF
          	at hudson.remoting.FastPipedInputStream.read(FastPipedInputStream.java:169)
          	at hudson.util.HeadBufferingStream.read(HeadBufferingStream.java:61)
          	at com.jcraft.jzlib.InflaterInputStream.fill(InflaterInputStream.java:175)
          	at com.jcraft.jzlib.InflaterInputStream.read(InflaterInputStream.java:106)
          	at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.read(TarArchiveInputStream.java:614)
          	at java.io.InputStream.read(InputStream.java:101)
          	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1792)
          	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769)
          	at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744)
          	at hudson.util.IOUtils.copy(IOUtils.java:40)
          	at hudson.FilePath.readFromTar(FilePath.java:2291)
          	... 13 more
          Caused by: java.io.IOException: Unexpected EOF
          	at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:99)
          	at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
          	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
          	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
          
          	at hudson.FilePath.copyRecursiveTo(FilePath.java:2217)
          	at jenkins.model.StandardArtifactManager.archive(StandardArtifactManager.java:61)
          	at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:219)
          	at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:75)
          	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
          	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
          	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:726)
          	at hudson.model.Build$BuildExecution.post2(Build.java:185)
          	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:671)
          	at hudson.model.Run.execute(Run.java:1766)
          	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          	at hudson.model.ResourceController.execute(ResourceController.java:98)
          	at hudson.model.Executor.run(Executor.java:408)
          Caused by: java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected EOF
          	at hudson.remoting.Request$1.get(Request.java:276)
          	at hudson.remoting.Request$1.get(Request.java:207)
          	at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
          	at hudson.FilePath.copyRecursiveTo(FilePath.java:2213)
          	... 12 more
          Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected EOF
          	at hudson.remoting.Request.abort(Request.java:297)
          	at hudson.remoting.Channel.terminate(Channel.java:844)
          	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
          Caused by: java.io.IOException: Unexpected EOF
          	at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:99)
          	at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
          	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
          	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
          Build step 'Archive the artifacts' changed build result to FAILURE
          

          and slave log has this:

          ERROR: Connection terminated
          java.io.IOException: Unexpected termination of the channel
          	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
          Caused by: java.io.EOFException
          	at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2335)
          	at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2804)
          	at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:802)
          	at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
          	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:40)
          	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
          	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
          
          Show
          jsposato John Sposato added a comment - I am seeing similar behavior on a CentOS 7 slave from Jenkins 1.633. Job errors during archiving with this: ERROR: Failed to archive artifacts: android/app/build/outputs/apk/*.apk java.io.IOException: java.io.IOException: Failed to extract /var/lib/jenkins/workspace/Wawa_Android_QA/transfer of 1 files at hudson.FilePath.readFromTar(FilePath.java:2301) at hudson.FilePath.copyRecursiveTo(FilePath.java:2210) at jenkins.model.StandardArtifactManager.archive(StandardArtifactManager.java:61) at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:219) at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:75) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779) at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:726) at hudson.model.Build$BuildExecution.post2(Build.java:185) at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:671) at hudson.model.Run.execute(Run.java:1766) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:408) Caused by: java.io.IOException: java.io.IOException: Unexpected EOF at hudson.remoting.FastPipedInputStream.read(FastPipedInputStream.java:169) at hudson.util.HeadBufferingStream.read(HeadBufferingStream.java:61) at com.jcraft.jzlib.InflaterInputStream.fill(InflaterInputStream.java:175) at com.jcraft.jzlib.InflaterInputStream.read(InflaterInputStream.java:106) at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.read(TarArchiveInputStream.java:614) at java.io.InputStream.read(InputStream.java:101) at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1792) at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769) at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744) at hudson.util.IOUtils.copy(IOUtils.java:40) at hudson.FilePath.readFromTar(FilePath.java:2291) ... 13 more Caused by: java.io.IOException: Unexpected EOF at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:99) at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) at hudson.FilePath.copyRecursiveTo(FilePath.java:2217) at jenkins.model.StandardArtifactManager.archive(StandardArtifactManager.java:61) at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:219) at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:75) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779) at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:726) at hudson.model.Build$BuildExecution.post2(Build.java:185) at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:671) at hudson.model.Run.execute(Run.java:1766) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:408) Caused by: java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected EOF at hudson.remoting.Request$1.get(Request.java:276) at hudson.remoting.Request$1.get(Request.java:207) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.FilePath.copyRecursiveTo(FilePath.java:2213) ... 12 more Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected EOF at hudson.remoting.Request.abort(Request.java:297) at hudson.remoting.Channel.terminate(Channel.java:844) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69) Caused by: java.io.IOException: Unexpected EOF at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:99) at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) Build step 'Archive the artifacts' changed build result to FAILURE and slave log has this: ERROR: Connection terminated java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50) Caused by: java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2335) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2804) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:802) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:40) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
          Hide
          jsposato John Sposato added a comment -

          we use the EC2 plugin and I was able to solve my problems by checking the option 'Connect by SSH Process'. The slaves have been 100% stable since that point.

          Show
          jsposato John Sposato added a comment - we use the EC2 plugin and I was able to solve my problems by checking the option 'Connect by SSH Process'. The slaves have been 100% stable since that point.
          Hide
          kerrhome Shannon Kerr added a comment -

          Thanks for the updated John. We do not use EC2 here. Still have this problem. All slaves except for the freescale one are stable. My team is blocked from using Jenkins for testing until this is fixed. I hope to see something soon.

          Show
          kerrhome Shannon Kerr added a comment - Thanks for the updated John. We do not use EC2 here. Still have this problem. All slaves except for the freescale one are stable. My team is blocked from using Jenkins for testing until this is fixed. I hope to see something soon.
          Hide
          danielbeck Daniel Beck added a comment -

          1.574 -> 1.609.3 is a bit too much of a jump to investigate the possible cause without having access to the affected machine – it's ~3.5 LTS releases!

          Please try to determine the specific release that broke this, and provide that information.

          Show
          danielbeck Daniel Beck added a comment - 1.574 -> 1.609.3 is a bit too much of a jump to investigate the possible cause without having access to the affected machine – it's ~3.5 LTS releases! Please try to determine the specific release that broke this, and provide that information.
          Hide
          kerrhome Shannon Kerr added a comment -

          This is now fixed when using Jenkins 1.642.1. I verified again that it was still broken with 1.609.3. I'm satisfied that the issue is resolved and I'm good to close the ticket. Daniel, are you good with me closing the ticket? I will not have time to isolate this down to a more-specific version of Jenkins.

          Show
          kerrhome Shannon Kerr added a comment - This is now fixed when using Jenkins 1.642.1. I verified again that it was still broken with 1.609.3. I'm satisfied that the issue is resolved and I'm good to close the ticket. Daniel, are you good with me closing the ticket? I will not have time to isolate this down to a more-specific version of Jenkins.
          Hide
          danielbeck Daniel Beck added a comment -

          Shannon Kerr Thanks for the update!

          Not an issue on current LTS, therefore resolving as Cannot Reproduce.

          Show
          danielbeck Daniel Beck added a comment - Shannon Kerr Thanks for the update! Not an issue on current LTS, therefore resolving as Cannot Reproduce.

            People

            • Assignee:
              Unassigned
              Reporter:
              kerrhome Shannon Kerr
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: