Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60528

jenkins can't allocate mesos nodes duet to lock

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • mesos-plugin
    • None
    • mesos 0.17 jenkins 2.138.4 centos 7

      There is a critical problem in mesos plugins 0.17. When a lot of job building in jenkins, jenkins master don't allocate mesos nodes and Mesos master produce a lot of outstanding offers.There is a critical problem in mesos plugins 0.17. When a lot of job building in jenkins, jenkins master don't allocate mesos nodes and Mesos master produce a lot of outstanding offers.We execute java command jstack -l 58(which is PID of jenkins master), as follow:

      "Thread-28255" #93222 prio=3 os_prio=0 tid=0x00007f89c0975432 nid=0xe7 runnable (0x00007f89c09432)         

       java.lang.Thread.state: RUNNABLE               

      at java.util.concurrent.LinkedBlockingQueue$LBQSpliterator.trySplit(LinkedBlockingQueue.java 890)             

        at java.util.stream.AbstractTask.compute(AbstractTask.java:297)

      at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)

      at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)

      at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401)

      ......

      Lockeked owable synchronizers:        

      -<0x000000006089c0975> (a java.util.concurrent.locks.ReentranLock$NonfairSync)   

      -<0x000000006089c0968> (a java.util.concurrent.locks.ReentranLock$NonfairSync)    "mesos-offer-processor-6berferd-32re-45fd-12erdf6578uy-0026" #103 prio=5 os_prio=0 tid=0x00007f89c0973451 nid=0xf7 waiting on condition[0x00007f89c0971000]  java.lang.Thread.State: WAITING (parking)       

      at sun.misc.Unsafe.park(Native Method)    - parking to wait for <0x000000006089c0975> (a java.util.concurrent.locks.ReentranLock$NonfairSync)    

      ......
      Thread mesos-offer-processor is always at WAITING status because Thread-28255 is own the ReetranLock.There, jenkins doesn't allocate the mesos nodes.The below code which in mesos plugin produce the Thread-28255.

      public void offerRescinded(SchedulerDriver driver, OfferId offerId)

      {      LOGGER.info("Rescinded offer"+offerId);          offerQueue.remove(offerId); }

      We use the solution which is remove the code "offerQueue.remove(offerId);". However, I think there should be better solution for question? 

            vinodkone Vinod Kone
            fengzhisha bing han
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: