Use BigDL to load the pre-trained Caffe model into the Spark program?

[zaiza...@gmail.com]

hello，I just learn about bigdl.Now i have a problem

how to use Use BigDL to load the pre-trained Caffe model into the Spark program?

thanks

[Yiheng Wang]

Hi

Please take a look at this example:

https://github.com/intel-analytics/BigDL/tree/master/spark/dl/src/main/scala/com/intel/analytics/bigdl/example/loadmodel

--
You received this message because you are subscribed to the Google Groups "BigDL User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigdl-user-group+unsubscribe@googlegroups.com.
To post to this group, send email to bigdl-user-group@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bigdl-user-group/5431a45f-8351-40f9-b5c8-f9d785f8105a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Yiheng Wang

yih...@gmail.com

[Yiheng Wang]

Hi

In a word, there's a Module.loadCaffe method to load caffe prototxt file and corresponding bin file to bigdl models.

You can see how it is used in the example(https://github.com/intel-analytics/BigDL/tree/master/spark/dl/src/main/scala/com/intel/analytics/bigdl/example/loadmodel). Specifically, it's at this line 

https://github.com/intel-analytics/BigDL/blob/master/spark/dl/src/main/scala/com/intel/analytics/bigdl/example/loadmodel/ModelValidator.scala#L112

--

Yiheng Wang

[zaiza...@gmail.com]

i dont know what is "-f <imagenet_folder> -o <output_folder> -p <cores_number>"?

imagenet_floder includes what?

i only have a caffe model,what should i get then?

thanks,i am a really new learner. 

在 2017年4月13日星期四 UTC+8下午1:36:39，yihengw写道：

Hi

In a word, there's a Module.loadCaffe method to load caffe prototxt file and corresponding bin file to bigdl models.

You can see how it is used in the example(https://github.com/intel-analytics/BigDL/tree/master/spark/dl/src/main/scala/com/intel/analytics/bigdl/example/loadmodel). Specifically, it's at this line 

https://github.com/intel-analytics/BigDL/blob/master/spark/dl/src/main/scala/com/intel/analytics/bigdl/example/loadmodel/ModelValidator.scala#L112

On Thu, Apr 13, 2017 at 1:32 PM, Yiheng Wang <yih...@gmail.com> wrote:

Hi

Please take a look at this example:

https://github.com/intel-analytics/BigDL/tree/master/spark/dl/src/main/scala/com/intel/analytics/bigdl/example/loadmodel

On Thu, Apr 13, 2017 at 12:52 PM, <zaiza...@gmail.com> wrote:

hello，I just learn about bigdl.Now i have a problem

how to use Use BigDL to load the pre-trained Caffe model into the Spark program?

thanks

--
You received this message because you are subscribed to the Google Groups "BigDL User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bigdl-user-gro...@googlegroups.com.
To post to this group, send email to bigdl-us...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigdl-user-group/5431a45f-8351-40f9-b5c8-f9d785f8105a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Yiheng Wang

yih...@gmail.com

--

Yiheng Wang

[Jason Dai]

The loadmodel example use the loaded model to test on the input imagenet images (which are supposed to be stored on HDFS).

You can use ImageNetSeqFileGenerator to generate the input imagenet images for the loadmodel example on HDFS (see https://github.com/intel-analytics/BigDL/tree/master/spark/dl/src/main/scala/com/intel/analytics/bigdl/models/inception#prepare-the-data), where -f <imagenet_folder> is where the original imagenet images are stored, and -o <output_folder> is where the output HDFS folder is, and -p cores_number is the number of cores on the local node.

Thanks,

-Jason

To unsubscribe from this group and stop receiving emails from it, send an email to bigdl-user-group+unsubscribe@googlegroups.com.
To post to this group, send email to bigdl-user-group@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bigdl-user-group/a6846b18-4f63-4fb6-9607-93fbcacce8b9%40googlegroups.com.

[zaiza...@gmail.com]

its still a error : "Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: /home/zzz/Caffe_model/dataset/JPEGImages/val is not valid"

JPEGImages(imagenet_floder) : original images

在 2017年4月14日星期五 UTC+8下午10:16:34，Jason Dai写道：

[翟应杰]

this means i need a Val in imagenet_folder?

Really confused

Help me......

<zaiza...@gmail.com>于2017年4月15日 周六上午12:32写道：

You received this message because you are subscribed to a topic in the Google Groups "BigDL User Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bigdl-user-group/gBCmfLhTh2c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bigdl-user-gro...@googlegroups.com.

To post to this group, send email to bigdl-us...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigdl-user-group/cd93d22c-b565-430d-b28e-8ddda42cbd7c%40googlegroups.com.

[Jennie Wang]

You can refer to https://github.com/intel-analytics/BigDL/tree/master/spark/dl/src/main/scala/com/intel/analytics/bigdl/models/inception#prepare-the-data. After you download the files(ILSVRC2012_img_train.tar and ILSVRC2012_img_val.tar) to your image_folder, and run the follow commands in your image_folder. There will be a val subdirectory under your image_folder. 

mkdir train
mv ILSVRC2012_img_train.tar train/
cd train
tar -xvf ILSVRC2012_img_train.tar
rm -f ILSVRC2012_img_train.tar
find . -name "*.tar" | while read CLASS_NAME ; do mkdir -p "${CLASS_NAME%.tar}"; tar -xvf "${CLASS_NAME}" -C "${CLASS_NAME%.tar}"; done
rm *.tar
cd ../
mkdir val
mv ILSVRC2012_img_val.tar val/
cd val
tar -xvf ILSVRC2012_img_val.tar
cat classes.lst | while read CLASS_NAME; do mkdir -p ${CLASS_NAME}; done
cat img_class.lst | while read PARAM; do mv ${PARAM/ n[0-9]*/} ${PARAM/ILSVRC*JPEG /}; done
rm ILSVRC2012_img_val.tar

And then use following script to create sequence file:

dist/bin/bigdl.sh --
java -cp dist/lib/bigdl-VERSION-jar-with-dependencies-and-spark.jar \
         com.intel.analytics.bigdl.models.utils.ImageNetSeqFileGenerator   \
    -f <imagenet_folder> -o <output_folder> -p <cores_number> -r -v

[zaiza...@gmail.com]

i know the meaning of "val",but i only find a "val.txt" in my dataset, not val subdirectory(not this example). so how to change the commands?

thanks

在 2017年4月15日星期六 UTC+8下午1:02:06，Jennie Wang写道：

[Jason Dai]

Hi,

Have you followed the exact steps at https://github.com/intel-analytics/BigDL/tree/master/spark/dl/src/main/scala/com/intel/analytics/bigdl/models/inception#prepare-the-data to prepare the input data?

In particular, if you have downloaded ILSVRC2012_img_val.tar, you can then:

mkdir val
mv ILSVRC2012_img_val.tar val/
cd val
tar -xvf ILSVRC2012_img_val.tar
cat classes.lst | while read CLASS_NAME; do mkdir -p ${CLASS_NAME}; done
cat img_class.lst | while read PARAM; do mv ${PARAM/ n[0-9]*/} ${PARAM/ILSVRC*JPEG /}; done 
rm ILSVRC2012_img_val.tar 

To unsubscribe from this group and stop receiving emails from it, send an email to bigdl-user-group+unsubscribe@googlegroups.com.
To post to this group, send email to bigdl-user-group@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigdl-user-group/09dedea8-50e7-46f1-b2cf-780da045e609%40googlegroups.com.