Where can I get documentation to get started with BigDL and the NotFruit project?

22 views
Skip to first unread message

Maurice Nsabimana

unread,
Aug 1, 2017, 1:50:02 PM8/1/17
to NotFruit
Hi everybody:

Where can I get documentation to get started with BigDL and the NotFruit project?

Thank you and looking forward to this!

Best,

/Maurice

Rashim Khadka

unread,
Aug 1, 2017, 3:24:02 PM8/1/17
to NotFruit
Hey Maurice,

You can refer to Intel's BigDL GitHub page: https://github.com/intel-analytics/BigDL-Tutorials, we also have detailed instructions provided in the following link that you can refer to https://docs.google.com/document/d/1WfBvoSY6gaNq_adA5qdbG7PpKa0LWe2aawXyDR576SM/edit?usp=sharing. 

I have uploaded a folder containing spark 2.1.0, BigDL version 0.2.0, start_notebook.sh(Execute start_notebook.sh in bash to launch jupyter), you can refer to this folder if you run into any problems and cross check the start_notebook.sh script. Once you have successfully launched the jupyter, you can run the MNIST example located in the tutorials subfolder. The bigdl jar file in this folder is only compatible with Mac, download the appropriate Linux version. 

Feel free to reach out if you have any further questions.

Cheers!

Rashim

Maurice Nsabimana

unread,
Aug 2, 2017, 11:47:48 AM8/2/17
to Rashim Khadka, NotFruit

Hi Rashim:

 

Thanks a bunch, I was able to launch Jupyter notebook by tweaking the startup script for Windows. Incidentally, it also seems to work with Python 3 but would require adjusting code syntax (as expected):

 

 

Will start playing with the examples…

 

Best,

 

/Maurice

--
You received this message because you are subscribed to the Google Groups "NotFruit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to notfruit+u...@googlegroups.com.
To post to this group, send email to notf...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/notfruit/ada5cb4c-d082-4142-840f-e47108275435%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Dave Nielsen

unread,
Aug 2, 2017, 1:09:50 PM8/2/17
to NotFruit
Nicely done. Next try to get some BigDL examples to run: 

In addition to the MNIST/Lenet example, you can try the SSD python notebook example here https://drive.google.com/open?id=0B7vkTZblCs9hdmtyVzRvRnY1a1U This zip file contains a model, ssd project jars, command file, readme and some sample images. You can follow the readme to download some necessary resources and start the notebook.

Dave


On Wednesday, August 2, 2017 at 8:47:48 AM UTC-7, Maurice Nsabimana wrote:

Hi Rashim:

 

Thanks a bunch, I was able to launch Jupyter notebook by tweaking the startup script for Windows. Incidentally, it also seems to work with Python 3 but would require adjusting code syntax (as expected):

 

 

Will start playing with the examples…

 

Best,

 

/Maurice

 

 

From: notf...@googlegroups.com [mailto:notfruit@googlegroups.com] On Behalf Of Rashim Khadka
Sent: Tuesday, August 01, 2017 3:24 PM
To: NotFruit <notf...@googlegroups.com>
Subject: Re: Where can I get documentation to get started with BigDL and the NotFruit project?

 

Hey Maurice,

 

You can refer to Intel's BigDL GitHub page: https://github.com/intel-analytics/BigDL-Tutorials, we also have detailed instructions provided in the following link that you can refer to https://docs.google.com/document/d/1WfBvoSY6gaNq_adA5qdbG7PpKa0LWe2aawXyDR576SM/edit?usp=sharing. 

 

I have uploaded a folder containing spark 2.1.0, BigDL version 0.2.0, start_notebook.sh(Execute start_notebook.sh in bash to launch jupyter), you can refer to this folder if you run into any problems and cross check the start_notebook.sh script. Once you have successfully launched the jupyter, you can run the MNIST example located in the tutorials subfolder. The bigdl jar file in this folder is only compatible with Mac, download the appropriate Linux version. 

 

Feel free to reach out if you have any further questions.

 

Cheers!

 

Rashim

 

 

On Tuesday, August 1, 2017 at 10:50:02 AM UTC-7, Maurice Nsabimana wrote:

Hi everybody:

 

Where can I get documentation to get started with BigDL and the NotFruit project?

 

Thank you and looking forward to this!

 

Best,

 

/Maurice

--
You received this message because you are subscribed to the Google Groups "NotFruit" group.

To unsubscribe from this group and stop receiving emails from it, send an email to notfruit+unsubscribe@googlegroups.com.

Maurice Nsabimana

unread,
Aug 4, 2017, 6:42:09 PM8/4/17
to NotFruit
Thank you both.

No success with BigDL on Windows. So I've switched to a Linux VM and was able to run all the examples (save for a few errors with the logistic regression and another notebook).

@Dave: I was able to get the SSD notebook to work though it is painfully slow on my host... I will try on a cloud VM.

Best,

/Maurice

To unsubscribe from this group and stop receiving emails from it, send an email to notfruit+u...@googlegroups.com.

Maurice Nsabimana

unread,
Aug 4, 2017, 8:13:32 PM8/4/17
to NotFruit
OMG, just found this link with Win64 BigDL packages: https://bigdl-project.github.io/master/#release-download/

Will try after dinner!!!

Mobile: +1 (202) 390-3677 | : @mutabazi | : @mutabazi
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
"Life is the first gift,
 love is the second,
 and understanding the third."
-- Marge Piercy
******************************************************************************

On Fri, Aug 4, 2017 at 6:42 PM, Maurice Nsabimana <muta...@gmail.com> wrote:
Boxbe This message is eligible for Automatic Cleanup! (muta...@gmail.com) Add cleanup rule | More info

Thank you both.

No success with BigDL on Windows. So I've switched to a Linux VM and was able to run all the examples (save for a few errors with the logistic regression and another notebook).

@Dave: I was able to get the SSD notebook to work though it is painfully slow on my host... I will try on a cloud VM.

Best,

/Maurice



On Wednesday, August 2, 2017 at 1:09:50 PM UTC-4, Dave Nielsen wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to notfruit+u...@googlegroups.com.


To post to this group, send email to notf...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/notfruit/ada5cb4c-d082-4142-840f-e47108275435%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "NotFruit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to notfruit+unsubscribe@googlegroups.com.
To post to this group, send email to notf...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/notfruit/5a58a48a-86a2-4bff-a1ca-f52639a50a9d%40googlegroups.com.

Dave Nielsen

unread,
Aug 4, 2017, 10:59:23 PM8/4/17
to Maurice Nsabimana, NotFruit

Sorry to hear about Windows. It's new and maybe not well debugged yet. I haven't even tried it yet. But congrats on getting SSD to work.

As for slowness, Cloud is good. Can you use World Bank cloud resources, or do you need Intel to provide some in aws? I have some as a part of the https://software.intel.com/bigdlcompute promo we are running. You are already signed-up

Dave

To unsubscribe from this group and stop receiving emails from it, send an email to notfruit+unsubscribe@googlegroups.com.

To post to this group, send email to notf...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Dave Nielsen
Technical Program Manager, BigDL, Deep Learning for Spark
Big Data Technologies, Intel Software
Organizer: SVBigData, SVDevOps, CloudCamp 
twitter davenielsen; linkedin dnielsen; fb: dcnielsen
skype: davenielsen; gtalk: dnielsen; mobile: 415-531-6674



Maurice Nsabimana

unread,
Aug 4, 2017, 11:55:05 PM8/4/17
to NotFruit
Relative success with BigDL (Win64) package:

The MNIST examples break by raising this exception when executing "trained_model = optimizer.optimize()" :

Py4JJavaError: An error occurred while calling o135.optimize.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 2.0 failed 1 times, most recent failure: Lost task 1.0 in stage 2.0 (TID 9, localhost, executor driver): java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Integer



SSD won't initialize on Windows (likely an issue with the Jar file)

Dave Nielsen

unread,
Aug 5, 2017, 12:32:54 AM8/5/17
to Maurice Nsabimana, NotFruit
That's great! I didn't know about the Win64 package. I'm guessing it is new with BigDL 0.2.0 release. 

That Py4JJavaError looks familiar. Look for a similar q&a answer in the BigDL Google Group/forum (im away from my computer)

Dave 

--
You received this message because you are subscribed to the Google Groups "NotFruit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to notfruit+u...@googlegroups.com.
To post to this group, send email to notf...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
--
Message has been deleted

Dave Nielsen

unread,
Aug 8, 2017, 5:05:09 PM8/8/17
to vegnonveg
Hi Maurice,

There are a couple of solutions mentioned about the Py4JJavaError in the BigDL group, but I can't tell if any of them worked: https://groups.google.com/forum/#!searchin/bigdl-user-group/py4jjavaerror

There's also mention of need for winutils.exe to avoid the Py4J error here: https://www.youtube.com/watch?v=t63PS3kiTTQ&feature=youtu.be&t=4m16s

Were you able to get past this error and run the MNIST example?

Dave

On Friday, August 4, 2017 at 9:32:54 PM UTC-7, Dave Nielsen wrote:
That's great! I didn't know about the Win64 package. I'm guessing it is new with BigDL 0.2.0 release. 

That Py4JJavaError looks familiar. Look for a similar q&a answer in the BigDL Google Group/forum (im away from my computer)

Dave 

On Fri, Aug 4, 2017 at 8:55 PM Maurice Nsabimana <muta...@gmail.com> wrote:
Relative success with BigDL (Win64) package:

The MNIST examples break by raising this exception when executing "trained_model = optimizer.optimize()" :

Py4JJavaError: An error occurred while calling o135.optimize.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 2.0 failed 1 times, most recent failure: Lost task 1.0 in stage 2.0 (TID 9, localhost, executor driver): java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Integer



SSD won't initialize on Windows (likely an issue with the Jar file)


On Friday, August 4, 2017 at 8:13:32 PM UTC-4, Maurice Nsabimana wrote:
OMG, just found this link with Win64 BigDL packages: https://bigdl-project.github.io/master/#release-download/

Will try after dinner!!!

Mobile: +1 (202) 390-3677 | : @mutabazi | : @mutabazi
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
"Life is the first gift,
 love is the second,
 and understanding the third."
-- Marge Piercy
******************************************************************************

On Fri, Aug 4, 2017 at 6:42 PM, Maurice Nsabimana  wrote:
This message is eligible for Automatic Cleanup! Add cleanup rule | More info
Reply all
Reply to author
Forward
0 new messages