| There is a critical problem in mesos plugins 0.17. When a lot of job building in jenkins, jenkins master don't allocate mesos nodes and Mesos master produce a lot of outstanding offers.There is a critical problem in mesos plugins 0.17. When a lot of job building in jenkins, jenkins master don't allocate mesos nodes and Mesos master produce a lot of outstanding offers.We execute java command jstack -l 58(which is PID of jenkins master), as follow: "Thread-28255" #93222 prio=3 os_prio=0 tid=0x00007f89c0975432 nid=0xe7 runnable (0x00007f89c09432) java.lang.Thread.state: RUNNABLE at java.util.concurrent.LinkedBlockingQueue$LBQSpliterator.trySplit(LinkedBlockingQueue.java 890) at java.util.stream.AbstractTask.compute(AbstractTask.java:297) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) ...... Lockeked owable synchronizers: -<0x000000006089c0975> (a java.util.concurrent.locks.ReentranLock$NonfairSync) -<0x000000006089c0968> (a java.util.concurrent.locks.ReentranLock$NonfairSync) "mesos-offer-processor-6berferd-32re-45fd-12erdf6578uy-0026" #103 prio=5 os_prio=0 tid=0x00007f89c0973451 nid=0xf7 waiting on condition[0x00007f89c0971000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x000000006089c0975> (a java.util.concurrent.locks.ReentranLock$NonfairSync) ......
Thread mesos-offer-processor is always at WAITING status because Thread-28255 is own the ReetranLock.There, jenkins doesn't allocate the mesos nodes.The below code which in mesos plugin produce the Thread-28255. public void offerRescinded(SchedulerDriver driver, OfferId offerId) { LOGGER.info("Rescinded offer"+offerId); offerQueue.remove(offerId); } We use the solution which is remove the code "offerQueue.remove(offerId);". However, I think there should be better solution for question? |
