Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50113

slave connected but offline

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Blocker
    • Resolution: Cannot Reproduce
    • Component/s: remoting
    • Labels:
    • Environment:
      Jenkins on ubuntu 16.04LTS, version 2.89.4
      slave on ubuntu 16.04LTS
    • Similar Issues:

      Description

      The connection with slave via ssh :

      Launch slave agent via SSH, Know hosts file Verification Strategy

      Slave log:

      [03/12/18 10:58:19] [SSH] Opening SSH connection to xxx:22.
      [03/12/18 10:58:19] [SSH] SSH host key matches key in Known Hosts file. Connection will be allowed.
      [03/12/18 10:58:19] [SSH] Authentication successful.
      [03/12/18 10:58:19] [SSH] The remote user's environment is: ...

      Expanded the channel window size to 4MB
      [03/12/18 10:58:19] [SSH] Starting slave process: cd "/home/dev/jenkins-smals-slave-3" && java  -jar slave.jar
      <===[JENKINS REMOTING CAPACITY]===>channel started
      Remoting version: 3.14
      This is a Unix agent

       

      But slave is still OFFLINE and the button 'Relaunch agent' is visible.

      The connection via terminal ssh is OK.

      Delete slave and atach them again in Jenkins did not work for us.

      We are really blocked now.

       

       

        Attachments

          Activity

          Hide
          step t stepanchuk added a comment -

          On Jenkins host in Jenkins home logs dir:

          4994 Mar 12 10:51 slave.log.4
          4871 Mar 12 11:04 slave.log.3
          4871 Mar 12 11:17 slave.log.2
          4871 Mar 12 11:33 slave.log.1
          4096 Mar 12 11:33 .
          3691 Mar 12 11:33 slave.log

          slave.log.4:

          [03/12/18 10:43:07] [SSH] Checking java version of java
          [03/12/18 10:43:08] [SSH] java -version returned 1.8.0_74.
          [03/12/18 10:43:08] [SSH] Starting sftp client.
          [03/12/18 10:43:08] [SSH] Remote file system root /home/dev/jenkins-smals-slave-3 does not exist. Will try to create it...
          [03/12/18 10:43:08] [SSH] Copying latest slave.jar...
          [03/12/18 10:43:08] [SSH] Copied 745,674 bytes.
          Expanded the channel window size to 4MB
          [03/12/18 10:43:08] [SSH] Starting slave process: cd "/home/dev/jenkins-smals-slave-3" && java  -jar slave.jar
          <===[JENKINS REMOTING CAPACITY]===>channel started
          Remoting version: 3.14
          This is a Unix agent
          Evacuated stdout
          Agent successfully connected and online
          Slave JVM has not reported exit code. Is it still running?
          ERROR: Connection terminated
          ha:////4EzVzpCvIp9IKGPBqYEFDw++6OK5hK90HQeQIBv/x9L2AAAAVx+LCAAAAAAAAP9b85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0mV3NWzufebKBsTA0NFEYMUmgZnCA1SyAABjCCFBQC2xNaiYAAAAA==java.io.EOFException
                  at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2335)
                  at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2804)
                  at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:802)
                  at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
                  at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
                  at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
                  at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
          Caused: java.io.IOException: Unexpected termination of the channel
                  at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
          [03/12/18 10:51:45] [SSH] Connection closed.

          -------------------------------------------------------------------------

          The 2 and 3 are the same, the last log: slave.log

          ...

          [03/12/18 11:33:28] [SSH] Checking java version of java
          [03/12/18 11:33:28] [SSH] java -version returned 1.8.0_74.
          [03/12/18 11:33:28] [SSH] Starting sftp client.
          [03/12/18 11:33:28] [SSH] Copying latest slave.jar...
          [03/12/18 11:33:28] [SSH] Copied 745,674 bytes.
          Expanded the channel window size to 4MB
          [03/12/18 11:33:28] [SSH] Starting slave process: cd "/home/dev/jenkins-smals-slave-3" && java  -jar slave.jar
          <===[JENKINS REMOTING CAPACITY]===>channel started
          Remoting version: 3.14
          This is a Unix agent

          ----------------------------------------------------------------------

          On the remote machine:

          java processes are running:

          bash -c cd "/home/dev/jenkins-smals-slave-3" && java  -jar slave.jar

           java -jar slave.jar

           

          Show
          step t stepanchuk added a comment - On Jenkins host in Jenkins home logs dir: 4994 Mar 12 10:51 slave.log.4 4871 Mar 12 11:04 slave.log.3 4871 Mar 12 11:17 slave.log.2 4871 Mar 12 11:33 slave.log.1 4096 Mar 12 11:33 . 3691 Mar 12 11:33 slave.log slave.log.4: [03/12/18 10:43:07] [SSH] Checking java version of java [03/12/18 10:43:08] [SSH] java -version returned 1.8.0_74. [03/12/18 10:43:08] [SSH] Starting sftp client. [03/12/18 10:43:08] [SSH] Remote file system root /home/dev/jenkins-smals-slave-3 does not exist. Will try to create it... [03/12/18 10:43:08] [SSH] Copying latest slave.jar... [03/12/18 10:43:08] [SSH] Copied 745,674 bytes. Expanded the channel window size to 4MB [03/12/18 10:43:08] [SSH] Starting slave process: cd "/home/dev/jenkins-smals-slave-3" && java  -jar slave.jar <=== [JENKINS REMOTING CAPACITY] ===>channel started Remoting version: 3.14 This is a Unix agent Evacuated stdout Agent successfully connected and online Slave JVM has not reported exit code. Is it still running? ERROR: Connection terminated ha:////4EzVzpCvIp9IKGPBqYEFDw++6OK5hK90HQeQIBv/x9L2AAAAVx+LCAAAAAAAAP9b85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0mV3NWzufebKBsTA0NFEYMUmgZnCA1SyAABjCCFBQC2xNaiYAAAAA==java.io.EOFException         at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2335)         at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2804)         at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:802)         at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)         at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)         at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)         at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63) Caused: java.io.IOException: Unexpected termination of the channel         at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77) [03/12/18 10:51:45] [SSH] Connection closed. ------------------------------------------------------------------------- The 2 and 3 are the same, the last log: slave.log ... [03/12/18 11:33:28] [SSH] Checking java version of java [03/12/18 11:33:28] [SSH] java -version returned 1.8.0_74. [03/12/18 11:33:28] [SSH] Starting sftp client. [03/12/18 11:33:28] [SSH] Copying latest slave.jar... [03/12/18 11:33:28] [SSH] Copied 745,674 bytes. Expanded the channel window size to 4MB [03/12/18 11:33:28] [SSH] Starting slave process: cd "/home/dev/jenkins-smals-slave-3" && java  -jar slave.jar <=== [JENKINS REMOTING CAPACITY] ===>channel started Remoting version: 3.14 This is a Unix agent ---------------------------------------------------------------------- On the remote machine: java processes are running: bash -c cd "/home/dev/jenkins-smals-slave-3" && java  -jar slave.jar  java -jar slave.jar  
          Hide
          jthompson Jeff Thompson added a comment -

          Can you collect more agent logs for the problematic agent? Increase the log level to more verbose, finer-grained messages. That might help demonstrate what is going on here.

          You can read about it here: https://github.com/jenkinsci/remoting/blob/master/docs/logging.md . In summary, if you add a java.util.logging properties file and then reference it via the `-loggingConfig` parameter to the agent. For example something like this: `-loggingConfig jenkins-logging.properties`.

          Show
          jthompson Jeff Thompson added a comment - Can you collect more agent logs for the problematic agent? Increase the log level to more verbose, finer-grained messages. That might help demonstrate what is going on here. You can read about it here: https://github.com/jenkinsci/remoting/blob/master/docs/logging.md  . In summary, if you add a java.util.logging properties file and then reference it via the `-loggingConfig` parameter to the agent. For example something like this: `-loggingConfig jenkins-logging.properties`.
          Hide
          jthompson Jeff Thompson added a comment -

          Closing for lack of sufficient diagnostics and information to reproduce along with no response for quite a while.

          Show
          jthompson Jeff Thompson added a comment - Closing for lack of sufficient diagnostics and information to reproduce along with no response for quite a while.

            People

            • Assignee:
              Unassigned
              Reporter:
              step t stepanchuk
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: