Wait in queue or retry when YARN cannot provide enough resource?

[DCh...@statestreet.com]

Hi,

When we run like 20 pipelines in parallel, and each one needs many resources, our YARN cluster cannot provide enough CPU/memory resources for all the pipelines. Then we find some pipelines will directly turn from “Starting” status to “Stopped”. What’s worse, some pipelines may firstly turn from “Starting” to “Running”, and then turn to “Failed” due to resources are not enough for its intermediate steps.

Is there a way to solve this? We were expecting that CDAP will have some mechanisms to manage requests in a queue if resources are not enough or retry if failed.

Thank you.

[Sreevatsan Raman]

Hi Chen,

If the YARN Application master for CDAP programs is in RUNNING state and does not get enough containers then the programs are shutdown after waiting for a certain period, which is controlled by twill.no.container.timeout which by default is 2 minutes.

If the YARN Application master for CDAP programs is in ACCEPTED state and does not get enough containers then the programs are shutdown after waiting for a certain period, which is controlled by app.program.max.start.seconds which by default is 5 minutes. If you know you will be oversubscribing your clusters, you can change those configurations to a larger value so that the programs will wait and run.

Note:
1. Changing this configuration will require a restart of CDAP master.
2. Both these configurations are specified at milli second precision.

Hope it helps.

Thanks,
Sree

--
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.
To post to this group, send email to cdap...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/a9cb370f-a491-4c2c-8d7a-ffcac37a9ca8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[DCh...@statestreet.com]

Thank you very much :)