Configuring Mesos plugin for allocating more than two executors

[Raghu Pallikonda]

Hello All,

   We have been running successfully a large fleet of Jenkins CIs. The version of the mesos plugin configured on the Jenkins instances is v0.7.0.  We have observed that the Mesos plugin is allocating two executors when two jenkins jobs are queued for execution, but when more than two jobs are queued then the executor allocation isn't very consistent (we have seen requests pipelined as well as executors spawned successfully )

The no. of executors allocation to the job/task queue isn't very consistent.  Jenkins command is configured to include the environment settings -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 to allow for more executor allocation as needed.

Could you please let us know if the above should suffice in having 2+ executors running successfully (when the mesos cluster have sufficient resources available ) and how we could triage the issue to fix any setup/configuration bug on our end.

If the 0.7.0 doesn't have the above feature, is that feature supported in any of the recent releases (0.8.0 - 0.10.0)? I went through the commit messages for the releases, but couldn't see that mentioned (Apologize in advance if I missed out on that).

Please let me know if I need to provide more information or need to clarify on the above.

Regards,

Raghu

[Manivannan]

Hi Raghu,

Jenkins not getting offers beyond 2 executors is not a problem with Mesos plugin. 

And the problem is very specific to your environment and is because of Mesos and the patch applied to normalize the frameworks when a new framework joins the cloud: https://github.com/maselvaraj/mesos/commit/e3107460b80c4f1fab9e04b0f72bee1527861805

If you remember there was an issue where Jenkins instances staying connected with mesos for a longer time lost their importance/priority in mesos cluster and stopped receiving offers. That is when we applied this patch so that all frameworks allocation counters are set to zero so that they compete equally. 

Thanks,

Mani

--
You received this message because you are subscribed to the Google Groups "Jenkins Mesos Plugin" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkins-meso...@googlegroups.com.
To post to this group, send email to jenkin...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkins-mesos/4f71b68d-d070-45c7-9609-2d2aa234f2c7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Raghu Pallikonda]

Hi Mani,

   Thank you for the quick reply. I recollect the discussion regarding normalizing the existing frameworks vis-a-vis new frameworks to ensure that mesos resource distribution fairness. 

I do not understand how the resource fairness patch affects/limit mesos to allocate only two executors. Could you help by going over the same. 

Also, please suggest on how we could solve both the issues at the same time ( more executors and resource fairness at the same time).

Thank you Mani for all the great help & support.

--Raghu

[Manivannan]

Hi Raghu,

I think you'll have to override the default Dominant resource fairness algorithm in Mesos by re-writing allocator module to your specific use case : http://mesos.apache.org/documentation/latest/allocation-module/

If you post the question in Mesos group you might be able to get some inputs on this.

Thanks,

Mani

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkins-mesos/017e28ed-6227-48a4-acc0-a6d984d29f35%40googlegroups.com.

[Raghu Pallikonda]

Hi Mani,

  Thank you for the allocator framework link. I will look into it. As you are aware, the cluster we are running also has the Aurora Framework registered to it. Aurora is able to get resources for multiple tasks that it is managing.  Are aurora framework requests different from the Jenkins mesos plugin (Mesos Framework) requests?  

Thanks,

Raghu

[Manivannan]

Hi Raghu,

Aurora would have a 'role' set. Slaves dedicated for aurora would  also have this 'role' set. Offers from these slaves would be always sent to Aurora no matter how old the Aurora framework: https://mesos.apache.org/documentation/latest/roles/

Thanks,

Mani

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkins-mesos/23e00e2b-ac22-43d9-9974-ccfa66601b78%40googlegroups.com.

[Manivannan]

I'm just talking about your specific environment. 'roles' are not set be default. 

[Raghu Pallikonda]

Hi Mani, 

    It is very helpful that you know our setup and environment very well.   Both the frameworks (Jenkins, Aurora) are getting offers, the issue here is the ability to get offers for more than 2 active tasks in Jenkins framework, where as the aurora framework is able to get offers.  We have verified that there are resources available in the cluster while the frameworks are waiting on the offers to be given by the Mesos master.

Could you please help me in understanding how the "roles" are associated with 2 active tasks issue.

Thanks,

Raghu

[sumesh kanayi]

Hi Raghu,

Were you able to get this isue fixed? Currently iam stuck with this issue