Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-57453

Invalid encoded sequence when connecting Windows Agent with Bitvise SSH Server

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Component/s: remoting, ssh-slaves-plugin
    • Labels:
      None
    • Environment:
    • Similar Issues:

      Description

      I'm trying to launch a Windows Jenkins agent via SSH. In our company we have successfully used Bitvise SSH Server for the recent years (mainly for SFTP file operations) so we would like to avoid installing a second SSH server on those machines.

      Unfortunately we are getting the following stack trace when launching the agent:

      Exception in thread "main" java.io.IOException: Invalid encoded sequence encountered: 00 00 00 00
      	at hudson.remoting.BinarySafeStream$1._read(BinarySafeStream.java:194)
      	at hudson.remoting.BinarySafeStream$1.read(BinarySafeStream.java:125)
      	at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2663)
      	at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2679)
      	at java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:3426)
      	at java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:3226)
      	at java.io.ObjectInputStream.readUTF(ObjectInputStream.java:1133)
      	at java.io.ObjectStreamClass.readNonProxy(ObjectStreamClass.java:768)
      	at java.io.ObjectInputStream.readClassDescriptor(ObjectInputStream.java:891)
      	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1857)
      	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
      	at hudson.remoting.Capability.read(Capability.java:170)
      	at hudson.remoting.ChannelBuilder.negotiate(ChannelBuilder.java:441)
      	at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:360)
      	at hudson.remoting.Launcher.main(Launcher.java:743)
      	at hudson.remoting.Launcher.runWithStdinStdout(Launcher.java:691)
      	at hudson.remoting.Launcher.run(Launcher.java:373)
      	at hudson.remoting.Launcher.main(Launcher.java:283)
      ERROR: Unexpected error in launching a agent. This is probably a bug in Jenkins.
      hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
      	at hudson.remoting.Request.abort(Request.java:340)
      	at hudson.remoting.Channel.terminate(Channel.java:1040)
      	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:94)
      	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to SERVER01234
      		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)
      		at hudson.remoting.Request.call(Request.java:202)
      		at hudson.remoting.Channel.call(Channel.java:956)
      		at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:626)
      		at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:432)
      		at hudson.plugins.sshslaves.SSHLauncher.startAgent(SSHLauncher.java:1034)
      		at hudson.plugins.sshslaves.SSHLauncher.access$500(SSHLauncher.java:128)
      		at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:868)
      		at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:833)
      		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      		at java.base/java.lang.Thread.run(Thread.java:834)
      Caused by: java.io.IOException: Unexpected termination of the channel
      	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
      Caused by: java.io.EOFException
      	at java.base/java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2763)
      	at java.base/java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3258)
      	at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:873)
      	at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:350)
      	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
      	at hudson.remoting.Command.readFrom(Command.java:140)
      	at hudson.remoting.Command.readFrom(Command.java:126)
      	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:36)
      	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
      [05/14/19 12:43:59] Launch failed - cleaning up connection
      ERROR: Connection terminated
      java.io.EOFException
      	at java.base/java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2763)
      	at java.base/java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3258)
      	at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:873)
      	at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:350)
      	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
      	at hudson.remoting.Command.readFrom(Command.java:140)
      	at hudson.remoting.Command.readFrom(Command.java:126)
      	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:36)
      	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
      Caused: java.io.IOException: Unexpected termination of the channel
      	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
      

      I've additionally tried it with Win32-OpenSSH (https://github.com/PowerShell/Win32-OpenSSH) and the agent started successfully.

      Any idea what goes wrong with the Bitvise SSH Server implementation? Is there any way to further debug this? I attached the full master logs (anonymized) to this issue.

        Attachments

          Activity

          Hide
          jthompson Jeff Thompson added a comment -

          This looks like a duplicate of JENKINS-44408, including the use of Bitvise. There are no solutions on that one, either. This one seems to be using Bitvise successfully, though, JENKINS-54394.

          The [Troubleshooting|https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md] page for ssh-slaves-plugin mentions this error and might be useful.

          This [StackOverflow post|https://stackoverflow.com/questions/44768139/unable-to-connect-to-slave-from-master-invalid-encoded-sequence-encountered], mentions the error, but it's not clear how it was resolved.

           

          Show
          jthompson Jeff Thompson added a comment - This looks like a duplicate of JENKINS-44408 , including the use of Bitvise. There are no solutions on that one, either. This one seems to be using Bitvise successfully, though, JENKINS-54394 . The [Troubleshooting| https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md ] page for ssh-slaves-plugin mentions this error and might be useful. This [StackOverflow post| https://stackoverflow.com/questions/44768139/unable-to-connect-to-slave-from-master-invalid-encoded-sequence-encountered ], mentions the error, but it's not clear how it was resolved.  
          Hide
          jansohn Robin Jansohn added a comment - - edited

          I don't think it's a duplicate of JENKINS-44408 as it's a completely different encoded sequence (00 00 00 00 vs 3C 3D 3D 3D 5B 48 55 44 53 4F 4E 20 ...). Same for the TROUBLESHOOTING site mentioned. This error I've reported also happens on an empty remote working directory.

          Also I think JENKINS-54394 is unrelated as the maintainer mentions JSch library in the comments which AFAICS is not used in ssh-slaves-plugin? The SSH connection is successfully established in my case but then it fails to start the channel for communication between the master and agent.

          So there's no way to further debug this?

          Show
          jansohn Robin Jansohn added a comment - - edited I don't think it's a duplicate of JENKINS-44408 as it's a completely different encoded sequence ( 00 00 00 00 vs 3C 3D 3D 3D 5B 48 55 44 53 4F 4E 20 ... ). Same for the TROUBLESHOOTING site mentioned. This error I've reported also happens on an empty remote working directory. Also I think JENKINS-54394 is unrelated as the maintainer mentions JSch library in the comments which AFAICS is not used in ssh-slaves-plugin? The SSH connection is successfully established in my case but then it fails to start the channel for communication between the master and agent. So there's no way to further debug this?
          Hide
          jthompson Jeff Thompson added a comment -

          I doubt the encode sequence values are a reliable differentiator. I searched for the message and there are very few instances of it. The few that exist are mostly connected with Windows and / or Bitvise. Those seem to be the commonalities. There has been a long history of connection issues generally with SSH servers on Windows.

          Yes, the ssh-slaves-plugin uses a forked version of the trilead ssh library instead of the jsch one. So that one may not be relevant.

          I'm afraid I don't have any good suggestions for troubleshooting / debugging this. It kind of looks like this has some relation to the SSH transport. The only instances of this error are connected to SSH agents and your experience using a different SSH server show distinct variations in results. The trick is probably to figure out what it is about the SSH transport that is introducing the problem. The logs you provided only show information at the upper, remoting level and are insufficient to get further. You could try using Wireshark or some other network capture tool. That gives such low-level results that it's hard to sort through it and isolate the specific issue. You might be able to figure something out from it, though.

          Show
          jthompson Jeff Thompson added a comment - I doubt the encode sequence values are a reliable differentiator. I searched for the message and there are very few instances of it. The few that exist are mostly connected with Windows and / or Bitvise. Those seem to be the commonalities. There has been a long history of connection issues generally with SSH servers on Windows. Yes, the ssh-slaves-plugin uses a forked version of the trilead ssh library instead of the jsch one. So that one may not be relevant. I'm afraid I don't have any good suggestions for troubleshooting / debugging this. It kind of looks like this has some relation to the SSH transport. The only instances of this error are connected to SSH agents and your experience using a different SSH server show distinct variations in results. The trick is probably to figure out what it is about the SSH transport that is introducing the problem. The logs you provided only show information at the upper, remoting level and are insufficient to get further. You could try using Wireshark or some other network capture tool. That gives such low-level results that it's hard to sort through it and isolate the specific issue. You might be able to figure something out from it, though.
          Hide
          ifernandezcalvo Ivan Fernandez Calvo added a comment -

          currently is not supported, I hope that when I have time to implement JENKINS-55363 you would use the ssh CLI native to make it. As a workaround, you can use launch an agent with a command on the master that allows you to use ssh CLI

          Show
          ifernandezcalvo Ivan Fernandez Calvo added a comment - currently is not supported, I hope that when I have time to implement JENKINS-55363 you would use the ssh CLI native to make it. As a workaround, you can use launch an agent with a command on the master that allows you to use ssh CLI
          Hide
          ifernandezcalvo Ivan Fernandez Calvo added a comment -

          Could you execute the following commands on the master and copy the results here? replace AGENT_USER with the user you use to connect to the agents, AGENT_HOST is the IP of the agent

           ssh AGENT_USER@AGENT_HOST -T java --version
           ssh AGENT_USER@AGENT_HOST -t java --version

          if the result is command not found try these ones

           ssh AGENT_USER@AGENT_HOST -T PATH_TOJAVA/java --version
           ssh AGENT_USER@AGENT_HOST -t PATH_TOJAVA/java --version
          Show
          ifernandezcalvo Ivan Fernandez Calvo added a comment - Could you execute the following commands on the master and copy the results here? replace AGENT_USER with the user you use to connect to the agents, AGENT_HOST is the IP of the agent ssh AGENT_USER@AGENT_HOST -T java --version ssh AGENT_USER@AGENT_HOST -t java --version if the result is command not found try these ones ssh AGENT_USER@AGENT_HOST -T PATH_TOJAVA/java --version ssh AGENT_USER@AGENT_HOST -t PATH_TOJAVA/java --version
          Hide
          jansohn Robin Jansohn added a comment - - edited

          Bitvise

          jenkins@1989e0ae98c2:/$ ssh jenkins-user@server01234.emea.mycompany.com -T C:/User/jdk8_x64/bin/java -version
          java version "1.8.0_201"
          Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
          Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)
          jenkins@1989e0ae98c2:/$
          
          jenkins@1989e0ae98c2:/$ ssh jenkins-user@server01234.emea.mycompany.com -t C:/User/jdk8_x64/bin/java -version
          java version "1.8.0_201"
          Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
          Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)
          Connection to server01234.emea.mycompany.com closed.
          jenkins@1989e0ae98c2:/$
          

          OpenSSH

          jenkins@1989e0ae98c2:/$ ssh jenkins-user@server01234.emea.mycompany.com -T C:/User/jdk8_x64/bin/java -version
          java version "1.8.0_201"
          Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
          Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)
          jenkins@1989e0ae98c2:/$
          
          jenkins@1989e0ae98c2:/$ ssh jenkins-user@server01234.emea.mycompany.com -t C:/User/jdk8_x64/bin/java -version
          Microsoft Windows [Version 10.0.14393]
          (c) 2016 Microsoft Corporation. All rights reserved.
          
          zf-world\jenkins-user@SERVER01234 C:\Users\jenkins-user>exitConnection to server01234.emea.mycompany.com closed.
          jenkins@1989e0ae98c2:/$
          

          The -t command behaves differently between the two SSH servers. OpenSSH simply opens the session on the remote side without executing the specified command (I have to manually disconnect from the session). Bitvise executes the command and closes the session again.

          Show
          jansohn Robin Jansohn added a comment - - edited Bitvise jenkins@1989e0ae98c2:/$ ssh jenkins-user@server01234.emea.mycompany.com -T C:/User/jdk8_x64/bin/java -version java version "1.8.0_201" Java(TM) SE Runtime Environment (build 1.8.0_201-b09) Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode) jenkins@1989e0ae98c2:/$ jenkins@1989e0ae98c2:/$ ssh jenkins-user@server01234.emea.mycompany.com -t C:/User/jdk8_x64/bin/java -version java version "1.8.0_201" Java(TM) SE Runtime Environment (build 1.8.0_201-b09) Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode) Connection to server01234.emea.mycompany.com closed. jenkins@1989e0ae98c2:/$ OpenSSH jenkins@1989e0ae98c2:/$ ssh jenkins-user@server01234.emea.mycompany.com -T C:/User/jdk8_x64/bin/java -version java version "1.8.0_201" Java(TM) SE Runtime Environment (build 1.8.0_201-b09) Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode) jenkins@1989e0ae98c2:/$ jenkins@1989e0ae98c2:/$ ssh jenkins-user@server01234.emea.mycompany.com -t C:/User/jdk8_x64/bin/java -version Microsoft Windows [Version 10.0.14393] (c) 2016 Microsoft Corporation. All rights reserved. zf-world\jenkins-user@SERVER01234 C:\Users\jenkins-user>exitConnection to server01234.emea.mycompany.com closed. jenkins@1989e0ae98c2:/$ The -t command behaves differently between the two SSH servers. OpenSSH simply opens the session on the remote side without executing the specified command (I have to manually disconnect from the session). Bitvise executes the command and closes the session again.
          Hide
          ifernandezcalvo Ivan Fernandez Calvo added a comment -

          I've checked where it fails and it happens when opening the remoting channel after copy the remoting.jar file, that's what bothers me it success to open the SSH connection and to copy the file by SFTP but fails when open the java channel to the stdout and to the stderr, I have to run some test in a local environment to see if I can make something to fix it.
          https://github.com/jenkinsci/ssh-slaves-plugin/blob/ssh-slaves-1.29.4/src/main/java/hudson/plugins/sshslaves/SSHLauncher.java#L1034

          Show
          ifernandezcalvo Ivan Fernandez Calvo added a comment - I've checked where it fails and it happens when opening the remoting channel after copy the remoting.jar file, that's what bothers me it success to open the SSH connection and to copy the file by SFTP but fails when open the java channel to the stdout and to the stderr, I have to run some test in a local environment to see if I can make something to fix it. https://github.com/jenkinsci/ssh-slaves-plugin/blob/ssh-slaves-1.29.4/src/main/java/hudson/plugins/sshslaves/SSHLauncher.java#L1034

            People

            • Assignee:
              jthompson Jeff Thompson
              Reporter:
              jansohn Robin Jansohn
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: