Looking for Virtual Machine that contains BigDL to create a cluster

[Hamza Col SAAIDIA]

Hello, 

In the last few weeks, I was trying to set up a Hadoop Spark BigDL cluster: master and two slaves to run BigDL machine learning algorithms, but it took me too long and I didn't figure the right way to do it, so I am looking for a prepared VM to use it instead.

Any help would be very handy

[Xin Qiu]

Sorry, we don't have  prepared  VM to create Hadoop/Spark cluster, we only have single node VM to try BigDL.

It looks like you are new to Hadoop/Spark, Hadoop/Spark cluster has two options, 

1. HDFS+Yarn cluster, spark installed on the client machine, (Most used, and recommended)

2. HDFS+Spark Standlone Cluster

BigDL is just a standard library for spark, only need to be installed on the client machine. 

There are a lots of guides for "How to set up a three-nodes hadoop cluster" and "How to run spark on yarn cluster" on Internet, when you run spark pi on yarn successfully, we can go to "How to run BigDL machine learning algorithms".

You can ask me, if you meet any problems.

And I'm wondering if you'd like to use BigDL's python or scala API? 

Bests,

-Xin

[Jason Dai]

You may run BigDL on any standard Hadoop/YARN cluster; see the tutorial at https://bigdl.readthedocs.io/en/latest/doc/Orca/Tutorial/yarn.html

Thanks,

-Jason

--
You received this message because you are subscribed to the Google Groups "User Group for BigDL" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigdl-user-gro...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bigdl-user-group/37739a01-51d0-483e-8766-506db2fdc926n%40googlegroups.com.

[Hamza Saaidia]

Thank you Mr. Xin, thank you Mr. Jason

We were able to create a cluster and run BigDL on it, we have tested the Mnist model existing in the official documentation as well and it worked fine. 

We are a university students, and our goal now is to train a custom Pytorch model -for Semantic Segmentation-,  but it seems too difficult for us since there is not enough examples showing how to do that weather using Pytorch or Keras api,  the only example I found the internet was for the famous Mnist, so I tried to adapt my python notebook the same way but I encountered many errors especially in dataset preparation since BigDL estimator doesn't work on the same dataset as the original model. 

I want to know weather you have any another examples like object detection algorithm, maybe we can inspire from them.

Thank you in advance, 

Looking forward to get a reply,

-Hamza