Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52293

Failing to accept fulfilled node requests

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: nodepool-agents-plugin
    • Labels:
      None
    • Environment:
      nodepool-agents 0.0.12
      Jenkins 2.121.1
      nodepool 3.1.0
    • Similar Issues:

      Description

      Nodepool fulfills the noderequest, then the request disappears in Jenkins and an exception appears in the logs.

       

      From nodepool: 

      ==> /var/log/nodepool/nodepool-launcher.log <==
      2018-07-01 01:30:27,120 INFO nodepool.PoolWorker.continuous-main: Assigning node request <NodeRequest {'nodes': [], 'state_time': 1530408622.0, 'id': '100-0000000003', 'declined_by': [], 'node_types': ['ubuntu-xenial'], 'state': 'requested', 'requestor': 'jenkins', 'stat': ZnodeStat(czxid=1628067, mzxid=1628067, ctime=1530408622671, mtime=1530408622671, version=0, cversion=0, aversion=0, ephemeralOwner=72606725090050086, dataLength=142, numChildren=0, pzxid=1628067), 'reuse': True}>
      
      ==> /var/log/nodepool/launcher-debug.log <==
      2018-07-01 01:30:27,123 DEBUG nodepool.driver.NodeRequestHandler[nodepool-10657-PoolWorker.continuous-main]: Accepting node request 100-0000000003
      2018-07-01 01:30:27,138 DEBUG nodepool.driver.NodeRequestHandler[nodepool-10657-PoolWorker.continuous-main]: Locked existing node 0000000003 for request 100-0000000003
      2018-07-01 01:30:27,148 DEBUG nodepool.driver.NodeRequestHandler[nodepool-10657-PoolWorker.continuous-main]: Fulfilled node request 100-0000000003
      2018-07-01 01:30:27,153 DEBUG nodepool.driver.NodeRequestHandler[nodepool-10657-PoolWorker.continuous-main]: Unlocked node 0000000003 for request 100-0000000003
      2018-07-01 01:30:27,167 DEBUG nodepool.PoolWorker.continuous-main: Removing handler for request 100-0000000003

       

      Jenkins logs:

      Jul 01, 2018 1:30:22 AM INFO com.rackspace.jenkins_nodepool.NodePool attemptProvisionWaiting on node to become available for task: ansible-lsn-site-func with label: nodepool-ubuntu-xenial with timeout: 500 seconds...
      Jul 01, 2018 1:30:27 AM WARNING com.rackspace.jenkins_nodepool.NodePool acceptNodesFailed to lock nodejava.util.ArrayList cannot be cast to java.lang.String
      java.lang.ClassCastException: java.util.ArrayList cannot be cast to java.lang.String
      	at com.rackspace.jenkins_nodepool.NodePoolNode.getNPType(NodePoolNode.java:39)
      	at com.rackspace.jenkins_nodepool.NodePoolNode.getJenkinsLabel(NodePoolNode.java:58)
      	at com.rackspace.jenkins_nodepool.NodePoolNode.getName(NodePoolNode.java:62)
      	at com.rackspace.jenkins_nodepool.NodePoolNode.toString(NodePoolNode.java:98)
      	at java.util.Formatter$FormatSpecifier.printString(Formatter.java:2886)
      	at java.util.Formatter$FormatSpecifier.print(Formatter.java:2763)
      	at java.util.Formatter.format(Formatter.java:2520)
      	at java.util.Formatter.format(Formatter.java:2455)
      	at java.lang.String.format(String.java:2940)
      	at com.rackspace.jenkins_nodepool.NodePool.acceptNodes(NodePool.java:222)
      	at com.rackspace.jenkins_nodepool.NodePool.attemptProvisionNode2(NodePool.java:531)
      	at com.rackspace.jenkins_nodepool.NodePool.attemptProvision(NodePool.java:485)
      	at com.rackspace.jenkins_nodepool.NodePool.provisionNode(NodePool.java:451)
      	at com.rackspace.jenkins_nodepool.NodePool.provisionNode(NodePool.java:425)
      	at com.rackspace.jenkins_nodepool.NodePools.provisionNode(NodePools.java:151)
      	at com.rackspace.jenkins_nodepool.NodePoolQueueListener.lambda$onEnterWaiting$0(NodePoolQueueListener.java:81)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      
      

       

      It persisted after stopping jenkins and nodepool, deleting all nodepool data in zookeeper, and then starting all services.

        Attachments

          Activity

          Hide
          logan Logan V added a comment -

          Back to working after downgrading to nodepool 3.0.1.

          Show
          logan Logan V added a comment - Back to working after downgrading to nodepool 3.0.1.
          Hide
          ddeal David Deal added a comment -

          The error is associated with the Zookeeper "type" field which previously contained a single string value. The recent updates to NodePool launcher appear to have changed the value for this field to be an ArrayList (more specifically, an array of strings).

          Show
          ddeal David Deal added a comment - The error is associated with the Zookeeper "type" field which previously contained a single string value. The recent updates to NodePool launcher appear to have changed the value for this field to be an ArrayList (more specifically, an array of strings).
          Hide
          ddeal David Deal added a comment - - edited

          nodepool-agents version 0.0.13 resolves this issue which was published 7/27/2018.

          Show
          ddeal David Deal added a comment - - edited nodepool-agents version 0.0.13 resolves this issue which was published 7/27/2018.
          Hide
          ddeal David Deal added a comment -

          Resolved in 0.0.13.

          Show
          ddeal David Deal added a comment - Resolved in 0.0.13.
          Show
          ddeal David Deal added a comment - Here's the PR  https://github.com/jenkinsci/nodepool-agents-plugin/pull/21

            People

            • Assignee:
              ddeal David Deal
              Reporter:
              logan Logan V
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: