Spark on yarn - sparkling-water Unsupported argument: (spark.dynamicAllocation.enabled,true) on CDH

Theyaa Matti

unread,

Jul 11, 2016, 7:06:50 PM7/11/16

to H2O Open Source Scalable Machine Learning - h2ostream

Hi All,
I am trying to run sparking-water on a Cloudera cluster 5.7, the issue I am facing is that I have to disable dynamic allocation in spark_on_yarn service in order to get Sparkling-water working. If I enable dynamic allocation, I get the following error. Unsupported argument: (spark.dynamicAllocation.enabled,true)

And I end up not able to launch.

Any ideas please?

mat...@0xdata.com

unread,

Jul 12, 2016, 4:54:27 AM7/12/16

to H2O Open Source Scalable Machine Learning - h2ostream

Hey Theya,

This is working as expected - H2O (and by proxy Sparkling Water) does not support dynamic allocation of nodes, once a cloud has been established the topology cannot be changed so we do not support spark.dynamicAllocation.enabled (it has to be set to false).

There's one more spark option we do not support due to our computation model and that is spark.speculation.

Regards,
Mateusz

Theyaa Matti

unread,

Jul 12, 2016, 9:51:59 AM7/12/16

to H2O Open Source Scalable Machine Learning - h2ostream

Thank you Mateuz, is there a guide on best practice for running sparkling water on cloudera and hortonworks?

Thanks

Theyaa.

dautk...@gmail.com

unread,

Oct 17, 2017, 11:57:50 PM10/17/17

to H2O Open Source Scalable Machine Learning - h2ostream

Does H2O has plans to add support for spark dynamic allocation?

Thank you.

Reply all

Reply to author

Forward