Can mr3 support yarn label scheduling?

[Carol Chapman]

Our yarn cluster has multiple tags, and different tags are used for different services.
However, I found that mr3 can only submit tasks to the default tag, which not only fails to meet the requirements, but also leads to some strange things.

As you can see, mr3 is used to submit a task that occupies 70 cores and 500g of memory, but its queue utilization rate has reached an amazing 7749%. The top is a normal Apache hive submitted task, which occupies 380 cores and 1.9tb of memory, but its queue proportion is normal.

As you can see, mr3 will only use the resources of the default partition.But the resources I give in the default partition are very small,I suspect that this will make mr3 unable to continue to apply for resources, because in my QA environment cluster, I do not use label scheduling, and mr3 has been working normally.

Issue I mentioned before:  MR3ApplicationMaster can not allocated container (google.com)

What do you think of this phenomenon?

[Sungwoo Park]

Hello,

mr3.queue.name in mr3-site.xml specifies the queue to which MR3 is submitted. Could you try changing mr3.queue.name and see what happens? In the current implementation of MR3 on Yarn, this is the code that sets ApplicationSubmissionContext:

    appSubmitContext.setApplicationType(MR3Constants.MR3_APPLICATION_TYPE)
    appSubmitContext.setApplicationName(clientName)
    appSubmitContext.setApplicationId(appId)
    appSubmitContext.setResource(amResource)
    queueName foreach { appSubmitContext.setQueue(_) }
    appSubmitContext.setCancelTokensWhenComplete(cancelDelegationTokensOnCompletion)
    appSubmitContext.setMaxAppAttempts(maxAppAttempts)
    appSubmitContext.setAMContainerSpec(containerLaunchContext)

Cheers,

--- Sungwoo

[Carol Chapman]

Here are some configuration information of my online cluster:

YARN QUEUE:
QUEUE: crowd   only  use default partition resource
QUEUE: default can use 15%~40% "normal" partition resource  and 1% ~40% default partition resource
QUEUE: EnterpriseQueue  can use 35%~40% "normal" partition resource  and 1% ~40% default partition resource
QUEUE: MrQueue can use 10%~40% "normal" partition resource  and 1%~40% default partition resource
QUEUE: STDQueue can use 40%~95% "normal" partition resource  and 1% ~40% default partition resource
QUEUE: yarn-system can use 15%~40% "normal" partition resource  and 0% ~ 20% default partition resource

Theoretically, crowd can use the resources of default partition and "normal" partition. When using Apache hive, the crowd queue can use the resources of the whole cluster, but when I use mr3, I can only apply for the resources of the queue without normal tag.But I can always request resources.

When I use a non crowd queue, because the resources I reserve for the default tag are very, very small, I find that I often cannot apply for any resources. The log feedback is that the resources requested from yarn are 0. Even if I applied for resources for the first time, after mr3 recycling the resources, I often can't apply for resources again.

[Sungwoo Park]

In the current implementation, MR3 does not specify application tags for its Yarn application.

We could add a new configuration key for specifying application tags for Hive on MR3.

Would this extension solve the problem in your use case? (I could upload MR3 1.4-snapshot anytime.)

Cheers,

--- Sungwoo

--
You received this message because you are subscribed to the Google Groups "MR3" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hive-mr3+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hive-mr3/d4da738c-5560-4d8a-83f0-ef2127e1d24dn%40googlegroups.com.

[Carol Chapman]

Yes, that sounds good

[Sungwoo Park]

I uploaded a pre-release of Hive-MR3 1.4.

https://github.com/mr3project/mr3-release/releases/tag/v1.4

You can use two new configurations in mr3-site.xml:

<property>
  <name>mr3.application.tags</name>
  <description>
    Comma-separated list of application tags for the MR3 job
  </description>
</property>

<property>
  <name>mr3.application.scheduling.properties.map</name>
  <description>
    Comma-separated list of scheduling properties for the MR3 job (e.g., `foo1=bar1,foo2=bar2`)
  </description>
</property>

Cheers,

--- Sungwoo

[Carol Chapman]

Thanks!