Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45295

Swarm Client should update labels on the fly when labelsFile changes

    Details

    • Type: New Feature
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: swarm-plugin
    • Labels:
    • Environment:
      jenkins 2.46.2
      Swarm Plugin 3.4
      java-1.7.0-openjdk-1.7.0.141-2.6.10.1.el7_3.x86_64

      everything started directly from command line, as bash script
    • Similar Issues:
    • Released As:
      3.17

      Description

      I want to dynamically inject labels but its always taking down the client.

      As I understand the -labelsFile parameter it should actually watch the file and pickup changes.

      Jul 04 15:46:26 n001 systemd[1]: Starting Jenkins Swarm Client...
      Jul 04 15:46:26 n001 run-node.sh[3258]: Jul 04, 2017 3:46:26 PM hudson.plugins.swarm.Client main
      Jul 04 15:46:26 n001 run-node.sh[3258]: INFO: Client.main invoked with: [-master http://n002.domain:8080 -executors 1 -labelsFile /opt/jenkins/2.46.2/labels -disableSslVerification -name n001.domain]
      Jul 04 15:46:26 n001 run-node.sh[3258]: Jul 04, 2017 3:46:26 PM hudson.plugins.swarm.SwarmClient <init>
      Jul 04 15:46:26 n001 run-node.sh[3258]: INFO: Loading labels from /opt/jenkins/2.46.2/labels...
      Jul 04 15:46:26 n001 run-node.sh[3258]: Jul 04, 2017 3:46:26 PM hudson.plugins.swarm.SwarmClient <init>
      Jul 04 15:46:26 n001 run-node.sh[3258]: INFO: Labels found in file: node pxe_vmhost
      Jul 04 15:46:26 n001 run-node.sh[3258]: Jul 04, 2017 3:46:26 PM hudson.plugins.swarm.SwarmClient <init>
      Jul 04 15:46:26 n001 run-node.sh[3258]: INFO: Effective label list: [node, pxe_vmhost]
      Jul 04 15:46:26 n001 run-node.sh[3258]: Jul 04, 2017 3:46:26 PM hudson.plugins.swarm.Client run
      Jul 04 15:46:26 n001 run-node.sh[3258]: INFO: Discovering Jenkins master
      Jul 04 15:46:26 n001 run-node.sh[3258]: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
      Jul 04 15:46:26 n001 run-node.sh[3258]: SLF4J: Defaulting to no-operation (NOP) logger implementation
      Jul 04 15:46:26 n001 run-node.sh[3258]: SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
      Jul 04 15:46:26 n001 run-node.sh[3258]: Jul 04, 2017 3:46:26 PM hudson.plugins.swarm.Client run
      Jul 04 15:46:26 n001 run-node.sh[3258]: INFO: Setting up LabelFileWatcher
      Jul 04 15:46:26 n001 run-node.sh[3258]: Jul 04, 2017 3:46:26 PM hudson.plugins.swarm.Client run
      Jul 04 15:46:26 n001 run-node.sh[3258]: INFO: Attempting to connect to http://n002.domain:8080/ c6885380-addd-48dd-a863-b608907014b9 with ID e03ef12c
      Jul 04 15:46:26 n001 run-node.sh[3258]: Jul 04, 2017 3:46:26 PM hudson.plugins.swarm.SwarmClient getCsrfCrumb
      Jul 04 15:46:26 n001 run-node.sh[3258]: SEVERE: Could not obtain CSRF crumb. Response code: 404
      Jul 04 15:46:26 n001 run-node.sh[3258]: Jul 04, 2017 3:46:26 PM hudson.remoting.jnlp.Main createEngine
      Jul 04 15:46:26 n001 run-node.sh[3258]: INFO: Setting up slave: n001.domain-e03ef12c
      Jul 04 15:46:26 n001 run-node.sh[3258]: Jul 04, 2017 3:46:26 PM hudson.remoting.jnlp.Main$CuiListener <init>
      Jul 04 15:46:26 n001 run-node.sh[3258]: INFO: Jenkins agent is running in headless mode.
      Jul 04 15:46:26 n001 run-node.sh[3258]: Jul 04, 2017 3:46:26 PM hudson.remoting.jnlp.Main$CuiListener status
      Jul 04 15:46:26 n001 run-node.sh[3258]: INFO: Locating server among [http://n002.domain:8080/]
      Jul 04 15:46:26 n001 run-node.sh[3258]: Jul 04, 2017 3:46:26 PM hudson.remoting.jnlp.Main$CuiListener status
      Jul 04 15:46:26 n001 run-node.sh[3258]: INFO: Agent discovery successful
      Jul 04 15:46:26 n001 run-node.sh[3258]: Agent address: n002.domain
      Jul 04 15:46:26 n001 run-node.sh[3258]: Agent port: 44617
      Jul 04 15:46:26 n001 run-node.sh[3258]: Identity: fc:af:71:b0:55:6f:02:fa:b3:cd:51:a1:52:92:ab:43
      Jul 04 15:46:26 n001 run-node.sh[3258]: Jul 04, 2017 3:46:26 PM hudson.remoting.jnlp.Main$CuiListener status
      Jul 04 15:46:26 n001 run-node.sh[3258]: INFO: Handshaking
      Jul 04 15:46:26 n001 run-node.sh[3258]: Jul 04, 2017 3:46:26 PM hudson.remoting.jnlp.Main$CuiListener status
      Jul 04 15:46:26 n001 run-node.sh[3258]: INFO: Connecting to n002.domain:44617
      Jul 04 15:46:26 n001 run-node.sh[3258]: Jul 04, 2017 3:46:26 PM hudson.remoting.jnlp.Main$CuiListener status
      Jul 04 15:46:26 n001 run-node.sh[3258]: INFO: Trying protocol: JNLP4-connect
      Jul 04 15:46:27 n001 run-node.sh[3258]: Jul 04, 2017 3:46:27 PM hudson.remoting.jnlp.Main$CuiListener status
      Jul 04 15:46:27 n001 run-node.sh[3258]: INFO: Remote identity confirmed: fc:af:71:b0:55:6f:02:fa:b3:cd:51:a1:52:92:ab:43
      Jul 04 15:46:28 n001 run-node.sh[3258]: Jul 04, 2017 3:46:28 PM hudson.remoting.jnlp.Main$CuiListener status
      Jul 04 15:46:28 n001 run-node.sh[3258]: INFO: Connected
      Jul 04 16:11:56 n001 run-node.sh[3258]: Jul 04, 2017 4:11:56 PM hudson.plugins.swarm.LabelFileWatcher run
      Jul 04 16:11:56 n001 run-node.sh[3258]: WARNING: WARNING: Normal process, soft label update failed. Unable to acquire labels from master to begin removal process., forcing swarm client reboot. This can be disruptive to Jenkins jobs. Check your swarm cli
      Jul 04 16:11:56 n001 run-node.sh[3258]: Jul 04, 2017 4:11:56 PM hudson.plugins.swarm.LabelFileWatcher hardLabelUpdate
      Jul 04 16:11:56 n001 run-node.sh[3258]: WARNING: NOTE: You do not have a -Djava.util.logging.config.file specified, but your labels file has changed. You will lose logging for the new client instance. Although the client will continue to work, you will 
      Jul 04 16:11:56 n001 run-node.sh[3258]: Jul 04, 2017 4:11:56 PM hudson.plugins.swarm.LabelFileWatcher run
      Jul 04 16:11:56 n001 run-node.sh[3258]: WARNING: LabelFileWatcher no longer running. Shutting down this instance of Swarm Client.
      

      I tried to debug this further by starting the swarm client with "-Djava.util.logging.config.file=/var/log/jenkins_node.log"

      then all I see is:

      [root@n001 ~]# systemctl status jenkins-node -l 
      ● jenkins-node.service - Jenkins Swarm Client
      Loaded: loaded (/opt/jenkins/2.46.2/jenkins-node.service; enabled; vendor preset: disabled)
      Active: inactive (dead) since Tue 2017-07-04 17:37:16 CEST; 34s ago
      Process: 12560 ExecStart=/opt/jenkins/2.46.2/run-node.sh (code=exited, status=0/SUCCESS)
      Main PID: 12560 (code=exited, status=0/SUCCESS)
      
      Jul 04 17:11:25 n001 systemd[1]: Started Jenkins Swarm Client.
      Jul 04 17:11:25 n001 systemd[1]: Starting Jenkins Swarm Client...
      Jul 04 17:11:25 n001 run-node.sh[12560]: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
      Jul 04 17:11:25 n001 run-node.sh[12560]: SLF4J: Defaulting to no-operation (NOP) logger implementation
      Jul 04 17:11:25 n001 run-node.sh[12560]: SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
      [root@n001 ~]# cat /var/log/jenkins_node.log 
      [root@n001 ~]#
      

      meaning I do not see anything when the labelsFile changes, just the process is gone after some seconds.

      loading the systems SLF4J using "java -classpath /usr/share/java/slf4j/slf4j-api.jar:/usr/share/java/slf4j/slf4j-jcl.jar ..." is also of no help.

      (inspecting the .pom file from https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/swarm-client/3.4/ makes me think slf4j is build in anyway)

       

       

       

       

        Attachments

          Issue Links

            Activity

            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            KK does not maintain this plugin anymore. Moving to unassigned to set the expectation

            Show
            oleg_nenashev Oleg Nenashev added a comment - KK does not maintain this plugin anymore. Moving to unassigned to set the expectation
            Hide
            basil Basil Crow added a comment -

            Can you try using -disableClientsUniqueId as a temporary workaround?

            Show
            basil Basil Crow added a comment - Can you try using -disableClientsUniqueId as a temporary workaround?
            Hide
            basil Basil Crow added a comment -

            Fixed in #110 and will be released in 3.17.

            Show
            basil Basil Crow added a comment - Fixed in #110 and will be released in 3.17.
            Hide
            basil Basil Crow added a comment -

            Released in 3.17.

            Show
            basil Basil Crow added a comment - Released in 3.17.

              People

              • Assignee:
                basil Basil Crow
                Reporter:
                ddb259 Stephan Ebelt
              • Votes:
                2 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: