RNN example

[alepb...@gmail.com]

Folks,

I have the same input shape as the one from the TextClassifier.scala:

val maxSequenceLength = 10
val embeddingDim = 100

val batching = Batching(param.batchSize, Array(maxSequenceLength, embeddingDim))
val trainingDataSet = DataSet.rdd(trainingRDD) -> batching
val valDataSet = DataSet.rdd(valRDD) -> batching

I'd like to traing an RNN to compute a classification task and I was trying something like this:

val hiddenSize = 40
val bpttTruncate = 4
val outputSize = 1000
val inputSize = 1000

val model_N = Sequential[Float]()
  .add(Recurrent[Float](hiddenSize, bpttTruncate)
  .add(RnnCell[Float](inputSize, hiddenSize))
  .add(Tanh[Float]()))
  .add(Linear[Float](hiddenSize, outputSize))
  .add(Linear(2, classNum))
  .add(LogSoftMax())

Where classNum is the number of labels I have. 

val optimizer = Optimizer(
  model = model_N,
  dataset = trainingDataSet,
  criterion = new CrossEntropyCriterion[Float]()
).asInstanceOf[DistriOptimizer[Float]].disableCheckSingleton()

val numEpochs = 5
optimizer.
  setState(state).
  setValidation(Trigger.everyEpoch, valDataSet, Array(new Top1Accuracy[Float])).
  setOptimMethod(new SGD[Float]()).
  setEndWhen(Trigger.maxEpoch(numEpochs)).
  optimize()

This is the error that Im getting.

java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: requirement failed: input should be a two dimension Tensor
 I think Im lost as to how reshape my data correctly. I would expect Batching to spit out miniBatches of [maxSequenceLength x embedding] but Im probably wrong. 

Any support would be much appreciated.

[shell...@gmail.com]

Hi,

Currently, the RNN model only supports one sample input, which means train the model using training samples one by one. The error is thrown when Recurrent.scala loads the input with dimemsion greater than 2. (One sentence is represented by a two-dimensional tensor)

The BigDL will support the RNN for minibatch training soon.

[alepb...@gmail.com]

Hi again,

I understand that the problem is my input being a 2d tensor so I tried to bring it back to one dimension and feed the RNN to perform a classification but I keep getting an exception.

Here is the code:

https://gist.github.com/whitebread/6da8f08f92332c1a6956de843a7addea

Here it's the exception:

java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: requirement failed

at java.util.concurrent.FutureTask.report(FutureTask.java:122)

at java.util.concurrent.FutureTask.get(FutureTask.java:192)

at com.intel.analytics.bigdl.optim.DistriOptimizer$$anonfun$4$$anonfun$7.apply(DistriOptimizer.scala:176)

at com.intel.analytics.bigdl.optim.DistriOptimizer$$anonfun$4$$anonfun$7.apply(DistriOptimizer.scala:176)

at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)

at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)

at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)

at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)

at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)

at scala.collection.AbstractTraversable.map(Traversable.scala:104)

at com.intel.analytics.bigdl.optim.DistriOptimizer$$anonfun$4.apply(DistriOptimizer.scala:176)

at com.intel.analytics.bigdl.optim.DistriOptimizer$$anonfun$4.apply(DistriOptimizer.scala:125)

at org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:89)

at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)

at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)

at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)

at org.apache.spark.scheduler.Task.run(Task.scala:85)

at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.IllegalArgumentException: requirement failed

at scala.Predef$.require(Predef.scala:207)

at com.intel.analytics.bigdl.tensor.DenseTensorMath$.addmv(DenseTensorMath.scala:603)

at com.intel.analytics.bigdl.tensor.DenseTensor.addmv(DenseTensor.scala:1204)

at com.intel.analytics.bigdl.nn.Linear.updateOutput(Linear.scala:70)

at com.intel.analytics.bigdl.nn.Linear.updateOutput(Linear.scala:29)

at com.intel.analytics.bigdl.nn.ParallelTable.updateOutput(ParallelTable.scala:36)

at com.intel.analytics.bigdl.nn.RnnCell.updateOutput(RNN.scala:66)

at com.intel.analytics.bigdl.nn.RnnCell.updateOutput(RNN.scala:28)

at com.intel.analytics.bigdl.nn.Recurrent.updateOutput(Recurrent.scala:47)

at com.intel.analytics.bigdl.nn.Recurrent.updateOutput(Recurrent.scala:26)

at com.intel.analytics.bigdl.nn.abstractnn.AbstractModule.forward(AbstractModule.scala:129)

at com.intel.analytics.bigdl.nn.Sequential.updateOutput(Sequential.scala:33)

at com.intel.analytics.bigdl.nn.abstractnn.AbstractModule.forward(AbstractModule.scala:129)

at com.intel.analytics.bigdl.optim.DistriOptimizer$$anonfun$4$$anonfun$5$$anonfun$apply$2.apply$mcI$sp(DistriOptimizer.scala:164)

at com.intel.analytics.bigdl.optim.DistriOptimizer$$anonfun$4$$anonfun$5$$anonfun$apply$2.apply(DistriOptimizer.scala:158)

at com.intel.analytics.bigdl.optim.DistriOptimizer$$anonfun$4$$anonfun$5$$anonfun$apply$2.apply(DistriOptimizer.scala:158)

at com.intel.analytics.bigdl.utils.ThreadPool$$anonfun$1$$anon$4.call(Engine.scala:119)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

Any help would be highly appreciated. Thanks,

Alessandro

Hi

[shell...@gmail.com]

Hi,

I bet the problem is that rnn model in BigDL mismatches your text classification model.

The rnn model in BigDL is an implementation of language model meaning that a sequence of inputs map to a sequence of outputs. ([x1, x2, x3, .., xn] -> [x2, x3, x4, .., xn+1], x_i is a vector.)

However, the rnn model for text classification requires a single or a sequence of input mapping to a single output. ([x] -> [y], x is a vector, y is a vector.)

As a result, the forward method in Recurrent layer will yield a sequence of hidden units given a sequence of inputs. The 2D tensor as an input represents one training sample. i.e. one sentence.

Each row correlates with the others. 

In your case, is the input a 1000-length vector [0.2 0.3 -0.2 .... 0.8], and the expected output is a 40-length vector ?

Could you please provide a detailed explanation of your recurrent steps? In text classification, each word is represented by a fixed length vector (1000?). Then the whole article would be

 wrapped as a matrix? In this way, you will only need the last hidden unit as the output generated from the Recurrent layer. Then an additional layer will be needed to extract the last hidden unit before the Linear layer.

Besides, in the toSample codes, it accepts (nRow = 1, nCol = 2) parameters. Does it mean that it will form the labeled data to a (2*500, 1) tuple?

Currently, batch training is not supported in Recurrent layer. Please feed the RNN one sample at a time.

[alepb...@gmail.com]

Hello there, thanks for the support so far.

Let me give some more context as you asked first:

In my training set I have sentences, each of them labeled (I have  a total of 34 different labels for now).

Each sentence is a concatenation of words, each of them vectorized by its word2vec representation (i.e. the word "dog" will be vectorized via word2vec_model.getVectors("dog")).

Each vector coming out from word2vec is 100 in size. 

Every sentence will have the same length , in other words I pad every sentence to be a fixed length vector.

Because my sentences are short I fix the maximum length of 1000 ( which means 10 words *100 each). If I have a sentence which is 3 words only the remaining 7*100 floats will be simply zeros.

So if for example I have "German shepherd" labeled as "Dog", the same sentence in my training set it will look something like this:

sentence_1 = [ 0.2,0.4,...x100, 0.3,0.2,...x100, 0.1,0.6,...x100, 0.0,0.0,0.0,...x700] , 1.0

where the first hundred floats are the word2vec representations of each word followed by the padding (700 zeros). The Float 1.0 at the end represents the label "Dog"

Hopefully this gives you some more clarity as of how my training set looks like. 

Having said so, I tried as you said to remodel the network and to reduce my batch size to 1 which threw this error:

java.lang.IllegalArgumentException: requirement failed: total batch size(1) should be at least two times of node number(1) * core number(1), please change your batch size

So that I changed to to 2. Please look at the gist for complete code:

https://gist.github.com/whitebread/0f78c8b7bf43f0522e59033fc23f3451

but I'm still getting the following exception:

java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: requirement failed

at java.util.concurrent.FutureTask.report(FutureTask.java:122)

at java.util.concurrent.FutureTask.get(FutureTask.java:192)

at com.intel.analytics.bigdl.optim.DistriOptimizer$$anonfun$4$$anonfun$7.apply(DistriOptimizer.scala:176)

at com.intel.analytics.bigdl.optim.DistriOptimizer$$anonfun$4$$anonfun$7.apply(DistriOptimizer.scala:176)

at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)

at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)

at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)

at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)

at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)

at scala.collection.AbstractTraversable.map(Traversable.scala:104)

at com.intel.analytics.bigdl.optim.DistriOptimizer$$anonfun$4.apply(DistriOptimizer.scala:176)

at com.intel.analytics.bigdl.optim.DistriOptimizer$$anonfun$4.apply(DistriOptimizer.scala:125)

at org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:89)

at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)

at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)

at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)

at org.apache.spark.scheduler.Task.run(Task.scala:85)

at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Please note the following observations:

val model_N = Sequential[Double]()
	.add(Recurrent[Double](hiddenSize, bpttTruncate)
	.add(RnnCell[Double](inputSize, hiddenSize))
	.add(Tanh[Double]()))
	// .add(Linear(2, classNum))
	.add(LogSoftMax())

is what I understood I needed to change to make the network perform the classification on my number of class (classNum=34). Im not sure I even need the Linear activation which I also tried but it ended up failing too.

Any support would be highly appreciated.Thanks!

[shell...@gmail.com]

Hi,

I understand your data format now.

Since the input sentence is represented by a 10*100 vector and label is one double, ([ 0.2,0.4,...x100, 0.3,0.2,...x100, 0.1,0.6,...x100, 0.0,0.0,0.0,...x700] , 1.0)

I would suggest setting the (nRows, nCols) = (10, 100) in your toSample class. Thus each sample will be a 10*100 tensor. The Recurrent model will read this sample row by row and do the recurrence.

The output of the Recurrent model will be a 10*40 tensor. (10 rows, each row with size 40)

What you need is to extract the last row from the output. Then pass it to the Linear(40, classNum) layer.

The parameters of the recurrent layer should be set as follows: (it will read 2D tensor and recurrent row by row)

hiddenSize = 40

inputSize = 100

Currently, there is no such layer executing the extraction function because the recurrent layer is initially designed for a language model. I have posted the issue onto the BigDL github and we will support it in a couple of days.

[Jason Dai]

I think you can:

1) Set  (nRows, nCols) = (10, 100) in toSample and do the recurrence using batch size 1 (for now, you can call Engine.init after calling SampleToBatch to walk-around the minimun batch size requirement, as the RNN example did), and the output of the recurrence is a 10x40 tensor

2) Add a reshape layer after the RNN to reshape the output of the recurrence to a 400 vector

3) Add a Linear(400, 34) layer after the reshape, followed by the LogSoftMax.

BTW, for the "java.util.concurrent.ExecutionException", you may search the logs from Spark tasks to see which layer is reporting errors.

Thanks,

-Jason

--
You received this message because you are subscribed to the Google Groups "BigDL User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigdl-user-group+unsubscribe@googlegroups.com.
To post to this group, send email to bigdl-user-group@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigdl-user-group/33e7ad0b-25b8-41ca-8e90-e1d3a458d72c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[alepb...@gmail.com]

Folks,

for some reason it is still failing. 

Here's the modifications after your suggestions (I didn't change the SampleToBatch implementation from the gist above):

val nrows = 10
val ncols = 100
val batchSize = 2
val trainSet = DataSet.rdd(trainingRDD) -> ToSample(nrows, ncols) -> SampleToBatch(batchSize)
val valSet = DataSet.rdd(valRDD) -> ToSample(nrows, ncols)  -> SampleToBatch(batchSize)

val hiddenSize = 40
val bpttTruncate = 4

val inputSize = 100
val classNum = 34

val model_N = Sequential[Double]()
  .add(Recurrent[Double](hiddenSize, bpttTruncate)
    .add(RnnCell[Double](inputSize, hiddenSize))
    .add(Tanh[Double]()))

  .add(Reshape(Array(400)))
  .add(Linear(400, classNum))
  .add(LogSoftMax())

Here's the error:

java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: requirement failed: input should be a two dimension Tensor

at java.util.concurrent.FutureTask.report(FutureTask.java:122)

at java.util.concurrent.FutureTask.get(FutureTask.java:192)

at com.intel.analytics.bigdl.optim.DistriOptimizer$$anonfun$4$$anonfun$7.apply(DistriOptimizer.scala:176)

at com.intel.analytics.bigdl.optim.DistriOptimizer$$anonfun$4$$anonfun$7.apply(DistriOptimizer.scala:176)

at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)

at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)

at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)

at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)

at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)

at scala.collection.AbstractTraversable.map(Traversable.scala:104)

at com.intel.analytics.bigdl.optim.DistriOptimizer$$anonfun$4.apply(DistriOptimizer.scala:176)

at com.intel.analytics.bigdl.optim.DistriOptimizer$$anonfun$4.apply(DistriOptimizer.scala:125)

at org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:89)

at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)

at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)

at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)

at org.apache.spark.scheduler.Task.run(Task.scala:85)

at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.IllegalArgumentException: requirement failed: input should be a two dimension Tensor

at scala.Predef$.require(Predef.scala:219)

at com.intel.analytics.bigdl.nn.Recurrent.updateOutput(Recurrent.scala:35)

at com.intel.analytics.bigdl.nn.Recurrent.updateOutput(Recurrent.scala:26)

at com.intel.analytics.bigdl.nn.abstractnn.AbstractModule.forward(AbstractModule.scala:129)

at com.intel.analytics.bigdl.nn.Sequential.updateOutput(Sequential.scala:33)

at com.intel.analytics.bigdl.nn.abstractnn.AbstractModule.forward(AbstractModule.scala:129)

at com.intel.analytics.bigdl.optim.DistriOptimizer$$anonfun$4$$anonfun$5$$anonfun$apply$2.apply$mcI$sp(DistriOptimizer.scala:164)

at com.intel.analytics.bigdl.optim.DistriOptimizer$$anonfun$4$$anonfun$5$$anonfun$apply$2.apply(DistriOptimizer.scala:158)

at com.intel.analytics.bigdl.optim.DistriOptimizer$$anonfun$4$$anonfun$5$$anonfun$apply$2.apply(DistriOptimizer.scala:158)

at com.intel.analytics.bigdl.utils.ThreadPool$$anonfun$1$$anon$4.call(Engine.scala:119)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

... 3 more

Jason, I can't initialize the Engine after the batching for how I designed the application so Ill stick to batchSize=2 for now.

Alessandro

To unsubscribe from this group and stop receiving emails from it, send an email to bigdl-user-gro...@googlegroups.com.
To post to this group, send email to bigdl-us...@googlegroups.com.

[Jason Dai]

The problem is due to bacthSize=2: currently RNN only support batchSize=1; we'll implement a workaround for you to use batchSize=1 soon.

Thanks,

-Jason

To unsubscribe from this group and stop receiving emails from it, send an email to bigdl-user-group+unsubscribe@googlegroups.com.
To post to this group, send email to bigdl-user-group@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bigdl-user-group/fcb3cfa4-78b7-42a3-9987-f8c1380efafa%40googlegroups.com.

[Jason Dai]

Hi Alessandro,

We just merged the fix (https://github.com/intel-analytics/BigDL/pull/427), so that you can train your model using batchSize=1; we are currently working on a fix to support batchSize>1 for RNN models.

Thanks,

-Jason

[alepb...@gmail.com]

Jason,

thanks for the fix. Although, if I compile the last merge I get an error with the Sample:

error: too many arguments for method copy: (other: com.intel.analytics.bigdl.dataset.Sample[Double])com.intel.analytics.bigdl.dataset.Sample[Double]

             buffer.copy(featureBuffer, labelBuffer,

What I did was recompiling my working version of BigDL with just the edits you made on Utils.scala ( https://github.com/intel-analytics/BigDL/commit/287a697fcb5b0a9f61e21d6e8a18eb1f6eab772a ). That allowed me to execute my code but I keep getting an error. 

Now the batch size is 1 so it should be fine but this is what the exception looks like:

java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: requirement failed: invalid size

at java.util.concurrent.FutureTask.report(FutureTask.java:122)

at java.util.concurrent.FutureTask.get(FutureTask.java:192)

at com.intel.analytics.bigdl.optim.DistriOptimizer$$anonfun$4$$anonfun$7.apply(DistriOptimizer.scala:176)

at com.intel.analytics.bigdl.optim.DistriOptimizer$$anonfun$4$$anonfun$7.apply(DistriOptimizer.scala:176)

at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)

at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)

at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)

at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)

at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)

at scala.collection.AbstractTraversable.map(Traversable.scala:104)

at com.intel.analytics.bigdl.optim.DistriOptimizer$$anonfun$4.apply(DistriOptimizer.scala:176)

at com.intel.analytics.bigdl.optim.DistriOptimizer$$anonfun$4.apply(DistriOptimizer.scala:125)

at org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:89)

at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)

at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)

at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)

at org.apache.spark.scheduler.Task.run(Task.scala:85)

at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.IllegalArgumentException: requirement failed: invalid size

at scala.Predef$.require(Predef.scala:219)

at com.intel.analytics.bigdl.tensor.DenseTensor.valueAt(DenseTensor.scala:513)

at com.intel.analytics.bigdl.nn.ClassNLLCriterion.updateOutput(ClassNLLCriterion.scala:44)

at com.intel.analytics.bigdl.nn.CrossEntropyCriterion.updateOutput(CrossEntropyCriterion.scala:40)

at com.intel.analytics.bigdl.nn.CrossEntropyCriterion.updateOutput(CrossEntropyCriterion.scala:32)

at com.intel.analytics.bigdl.nn.abstractnn.AbstractCriterion.forward(AbstractCriterion.scala:43)

at com.intel.analytics.bigdl.optim.DistriOptimizer$$anonfun$4$$anonfun$5$$anonfun$apply$2.apply$mcI$sp(DistriOptimizer.scala:165)

at com.intel.analytics.bigdl.optim.DistriOptimizer$$anonfun$4$$anonfun$5$$anonfun$apply$2.apply(DistriOptimizer.scala:158)

at com.intel.analytics.bigdl.optim.DistriOptimizer$$anonfun$4$$anonfun$5$$anonfun$apply$2.apply(DistriOptimizer.scala:158)

at com.intel.analytics.bigdl.utils.ThreadPool$$anonfun$1$$anon$4.call(Engine.scala:119)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

 

I tried to put the same state/optimizer you guys use in the RNN example as it looks like the error is coming from the updateOutput. 

The model remains the following:

val nrows = 10

val ncols = 100

val batchSize = 1

val trainSet = DataSet.rdd(trainingRDD) -> ToSample(nrows, ncols) -> SampleToBatch(batchSize)

val valSet = DataSet.rdd(valRDD) -> ToSample(nrows, ncols)  -> SampleToBatch(batchSize)

val hiddenSize = 40

val bpttTruncate = 4

val inputSize = 100

val classNum = 34

val model_N = Sequential[Double]()

  .add(Recurrent[Double](hiddenSize, bpttTruncate)

    .add(RnnCell[Double](inputSize, hiddenSize))

    .add(Tanh[Double]()))

  .add(Reshape(Array(400)))

  .add(Linear(400, classNum))

  .add(LogSoftMax())

So far I tried both:

1)

val learningRate: Double = 0.1

val momentum: Double = 0.0

val weightDecay: Double = 0.0

val dampening: Double = 0.0

val state = {

  T("learningRate" -> learningRate,

    "momentum" -> momentum,

    "weightDecay" -> weightDecay,

    "dampening" -> dampening)

}

val optimizer = Optimizer(

  model = model_N,

  dataset = trainSet,

  criterion = new CrossEntropyCriterion[Double]()

).asInstanceOf[DistriOptimizer[Double]].disableCheckSingleton()

val numEpochs = 5

optimizer

  .setValidation(Trigger.everyEpoch, valSet, Array(new Loss[Double]))

  .setState(state)

  .setEndWhen(Trigger.maxEpoch(numEpochs))

  .optimize() 

2)

val state = T("learningRate" -> 0.01, "learningRateDecay" -> 0.0002)

val optimizer = Optimizer(

  model = model_N,

  dataset = trainSet,

  criterion = new ClassNLLCriterion[Double]()

).asInstanceOf[DistriOptimizer[Double]].disableCheckSingleton()

val numEpochs = 5

optimizer.

  setState(state).

  setValidation(Trigger.everyEpoch, valSet, Array(new Top1Accuracy[Double])).

  setOptimMethod(new Adagrad[Double]()).

  setEndWhen(Trigger.maxEpoch(numEpochs)).

  optimize()

Sorry if this is becoming painful for you as well and thanks for the support.

Alessandro

[shell...@gmail.com]

Hi,

Please use buffer.set() instead.

[Jason Dai]

Hi Alessandro,

Thanks for the detailed information; we have looked into the problem and finally got to the bottom of the problem; here are two code snippets (https://gist.github.com/zhangxiaoli73/5dc4ab0ca468c3e83156f6782e7d21e8 and https://gist.github.com/zhangxiaoli73/6802ff39e0ac5c3f0640041d1b34a6ca) that should work with the latest BigDL code on github.

1) Each input record is a tuple of feature (a 10x100 tensor) and label (a double), and we should change ToSample.apply to something like:

       ...

       if (labelBuffer == null) { 

           labelBuffer = new Array[Double](1) 

       }

       ...

       buffer.set(featureBuffer, labelBuffer, Array(nRows, nCols), Array(1))

2) As mentioned before, currently the Recurrent layer implements a language model and only works with batchSize=1; as a result, it will output a 10x40 tensor for your model and ignores the batch size information, which however is a mismatch for your input classification labels (a batchSizex1 tensor for each MiniBatch). We just fixed the Recurrent layer to keep the batch size information (see https://github.com/intel-analytics/BigDL/pull/430), and replaced the Reshape layer with a Select layer which can extract the output of the last element (a vector of size 40) in the Recurrent layer for classification:

     val model_N = Sequential[Double]()

       .add(Recurrent[Double](hiddenSize, bpttTruncate)

         .add(RnnCell[Double](inputSize, hiddenSize))

         .add(Tanh[Double]()))

         .add(Select(2, 10))

         // .add(Reshape(Array(400)))

        .add(Linear(40, classNum))

        .add(LogSoftMax())

Hope this helps; we are currently working on a fix to support batchSize>1 in the Recurrent layer.

Thanks,

-Jason

To unsubscribe from this group and stop receiving emails from it, send an email to bigdl-user-group+unsubscribe@googlegroups.com.
To post to this group, send email to bigdl-user-group@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigdl-user-group/c4a6c9a2-101c-4552-bfc8-2c1dd1bec201%40googlegroups.com.

[alepb...@gmail.com]

Hey Jason,

thanks for the support. The RNN is indeed working now with the latest fixes you made.

I saw the pull request for supporting batch size greater than 1 which is really good news. 

As of now I am not  able to converge to good accuracy but again, the training takes forever and the number of epochs has to be significant to make the network to work properly.  

As soon as you guys merge the version mentioned above Ill be happy to share my results.

Thanks,

Alessandro

[Jason Dai]

Hi Alessandro,

We just merged the full request for supporting batchSize>1 in RNN, and have tested the example above using batchSize=448 on 4 nodes.

Thanks,

-Jason

To unsubscribe from this group and stop receiving emails from it, send an email to bigdl-user-group+unsubscribe@googlegroups.com.
To post to this group, send email to bigdl-user-group@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bigdl-user-group/ede72c02-83e8-4bb0-b73e-40a0693696b1%40googlegroups.com.

[alepb...@gmail.com]

Hey Jason,

the batch size works just great so first of all thanks.

Although, being able to work with more data raised another couple of issues. 

The batching process seems to be incredibly slowly. It might be my lack of understanding in how the iterators/transformers work in BigDL but if I look at this line:

val trainSet = DataSet.rdd(trainingRDD) -> ToSample(nrows, ncols) -> SampleToBatch(batchSize)

(please refer to the previous gists to have the exact implementation of ToSample)

What I'm trying to do is simply batching my data in going from RDD[Array[Double],Double] to AbstractDataSet[com.intel.analytics.bigdl.dataset.MiniBatch[Double], _]. 

As long as I work with a small dataset(~25K) this takes about 10 minutes but when I try with a 10 times bigger dataset (250K) it takes about 3 hours to finish(!).

On a related note, I was also trying to save/load the model and test it right after but again, there is something wrong on the way I prep the data. More specifically:

val path_savedModel = "model_test"
val loaded_model = Module.load[Double](path_savedModel)

Recalling trainSet from above, shouldn't I be able to predict it doing something like this:

loaded_model.forward(trainSet)

but I get this exception:

 found   : com.intel.analytics.bigdl.dataset.AbstractDataSet[com.intel.analytics.bigdl.dataset.MiniBatch[Double],_$5] where type _$5
 required: com.intel.analytics.bigdl.nn.abstractnn.Activity
And Im not sure how to go from AbstractDataSet to an Activity.

Thanks for the support.

[Zhang, Cherry]

Hi Alessandro,

 

I just use script from https://gist.github.com/zhangxiaoli73/6802ff39e0ac5c3f0640041d1b34a6ca, and do some test.

Spark local, 1 node and 1 core, 378K sentences, batchSize=10, epoch=5, just take 36.58 minutes to finish train and test.

So, please describe your question more clearly and give your code if possible.

 

Go from AbstractDataSet to an Activity, you can use trainset.toDistributed.data(train=false) to get RDD[MiniBatch], example code :

val path_saveModel = "model_test"

val loaded_model = Module.load[Double](path_saveModel)

val tmpRDD = trainSet.toDistributed().data(train = false) //if not for train, please set train = false

val tmpOutput = tmpRDD.mapPartitions(dataIter => {

  dataIter.map(batch => {

    val input = batch.data

    loaded_model.forward(input).toTensor[Double]

    })

}).collect()

 

If you want to test your model, you can also refer to some test example code (such as: https://github.com/intel-analytics/BigDL/tree/master/dl/src/main/scala/com/intel/analytics/bigdl/models/lenet)

 

Thanks,

 

Cherry

To view this discussion on the web visit https://groups.google.com/d/msgid/bigdl-user-group/a10ffe19-96f5-45aa-a610-47fa37f43d2a%40googlegroups.com.

[Jason Dai]

Hi Alessandro,

I wonder if you can share more details on what exactly you mean by "batching process seems to be incredibly slowly"? The transformers (ToSample and SampleToBatch) are built on RDD operations (when running on Spark), which are lazily executed just as other RDD operations. "val trainSet = DataSet.rdd(trainingRDD) -> ToSample(nrows, ncols) -> SampleToBatch(batchSize)" will only load the data, randomly shuffle it over network and then cache it in memory. The actual batching (including both ToSample and SampleToBatch) only happens during training (Optimizer.optimize).

If your data loading (and shuffle) is slow - I wonder where your data are stored and whether there are network I/Os. And if the training is slow - I wonder what the throughput BigDL reports for each iterations for your training (we have seen >800 records/sec using 1 core in Spark local).

Thanks,

-Jason

To unsubscribe from this group and stop receiving emails from it, send an email to bigdl-user-group+unsubscribe@googlegroups.com.
To post to this group, send email to bigdl-user-group@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigdl-user-group/a10ffe19-96f5-45aa-a610-47fa37f43d2a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "BigDL User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bigdl-user-group+unsubscribe@googlegroups.com.
To post to this group, send email to bigdl-user-group@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigdl-user-group/1E524C3F99D09F48A87AADC7CDAE45A838560613%40shsmsx102.ccr.corp.intel.com.

[alepb...@gmail.com]

Jason, 

thanks for the explanation and sorry for the lack of details. As you identified already, the problem that Im facing is most likely due to the data loading part. 

Im not sure what I could have done wrong. 

Im using Scala 2.11 and Spark 2.0 on my local machine. 

The csv Im loading is also on my local machine and Im re-attaching the loading code for clarity:

val nodeNum = 1
val coreNum = 8
val sc = new SparkContext(Engine.init(nodeNum, coreNum, true).get.setMaster("local[*]").setAppName("test").set("spark.task.maxFailures", "1"))
val sqlContext = new SQLContext(sc)
val path = "~/Desktop/PATH_TSV.tsv"
val data = sqlContext.read.format("com.databricks.spark.csv").option("header", "true").option("delimiter", "\t").load(path)

// some more processing

val vectorizedRdd :RDD[(Array[Double], Double)] = labeled.select("labels","vectors").rdd.map(r => (r(1).asInstanceOf[WrappedArray[Double]].toArray,r(0).asInstanceOf[Double] + 1.0))
val Array(trainingRDD, valRDD) = vectorizedRdd.randomSplit(Array(trainingSplit, 1 - trainingSplit))

val nrows = 10
val ncols = 100

val batchSize = 64

val trainSet = DataSet.rdd(trainingRDD) -> ToSample(nrows, ncols) -> SampleToBatch(batchSize)
val valSet = DataSet.rdd(valRDD) -> ToSample(nrows, ncols)  -> SampleToBatch(batchSize)

// which is where it takes about 2hrs to load ~250K rows

This is a couple of snapshot of the throughput during the same training:

7-02-08 14:14:31 INFO  DistriOptimizer$:241 - [Epoch 4 157248/244390][Iteration 13915][Wall Clock 2789.300859796s] Train 64 in 0.194042336seconds. Throughput is 329.8249305759749 records/second. Loss is 1.755370063192032.

7-02-08 14:37:17 INFO  DistriOptimizer$:241 - [Epoch 6 89472/244390][Iteration 20494][Wall Clock 4155.918198255s] Train 64 in 0.21744152seconds. Throughput is 294.33201165996263 records/second. Loss is 1.9625739991949165

Do you see anything I am doing wrong?

On a side note:

Im not able to access port 4040 to monitor spark jobs. Maybe this is related to the issue above. In any case Im attaching the output when I first launch the sparkContext:

2017-02-08 11:18:43 INFO  ThreadPool$:87 - Set mkl threads to 1 on thread 1

2017-02-08 11:18:43 WARN  Engine$:344 - Invalid env setting. Please use bigdl.sh set the env. For spark application, please use Engine.sparkConf() to initialize your sparkConf

2017-02-08 11:18:43 INFO  ThreadPool$:87 - Set mkl threads to 1 on thread 1

2017-02-08 11:18:43 WARN  SparkConf:66 - The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.

2017-02-08 11:18:43 INFO  SparkContext:54 - Running Spark version 2.0.0

2017-02-08 11:18:43 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

2017-02-08 11:18:44 WARN  Utils:66 - Your hostname, XXXXXXXX resolves to a loopback address: 127.0.0.1; using 192.168.1.XXX instead (on interface en0)

2017-02-08 11:18:44 WARN  Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address

2017-02-08 11:18:44 INFO  SecurityManager:54 - Changing view acls to: user

2017-02-08 11:18:44 INFO  SecurityManager:54 - Changing modify acls to: user

2017-02-08 11:18:44 INFO  SecurityManager:54 - Changing view acls groups to: 

2017-02-08 11:18:44 INFO  SecurityManager:54 - Changing modify acls groups to: 

2017-02-08 11:18:44 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(user); groups with view permissions: Set(); users  with modify permissions: Set(user); groups with modify permissions: Set()

2017-02-08 11:18:44 INFO  Utils:54 - Successfully started service 'sparkDriver' on port 49347.

2017-02-08 11:18:44 INFO  SparkEnv:54 - Registering MapOutputTracker

2017-02-08 11:18:44 INFO  SparkEnv:54 - Registering BlockManagerMaster

2017-02-08 11:18:44 INFO  DiskBlockManager:54 - Created local directory at /private/var/folders/dy/13g7lrsj31d34ccjfm33hqcsqr4dwy/T/blockmgr-0ed2f0ac-5e71-4148-b68e-ae0949e041c1

2017-02-08 11:18:44 INFO  MemoryStore:54 - MemoryStore started with capacity 8.4 GB

2017-02-08 11:18:44 INFO  SparkEnv:54 - Registering OutputCommitCoordinator

2017-02-08 11:18:44 INFO  log:186 - Logging initialized @18330ms

2017-02-08 11:18:44 INFO  Server:327 - jetty-9.2.z-SNAPSHOT

2017-02-08 11:18:44 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@495e8a3{/jobs,null,AVAILABLE}

2017-02-08 11:18:44 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@6a7aa675{/jobs/json,null,AVAILABLE}

2017-02-08 11:18:44 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@6eded11a{/jobs/job,null,AVAILABLE}

2017-02-08 11:18:44 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@602a3237{/jobs/job/json,null,AVAILABLE}

2017-02-08 11:18:44 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@4b511e61{/stages,null,AVAILABLE}

2017-02-08 11:18:44 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@74a74070{/stages/json,null,AVAILABLE}

2017-02-08 11:18:44 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@6c6919ff{/stages/stage,null,AVAILABLE}

2017-02-08 11:18:44 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@5de335cf{/stages/stage/json,null,AVAILABLE}

2017-02-08 11:18:44 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@1e029a04{/stages/pool,null,AVAILABLE}

2017-02-08 11:18:44 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@50e8ed74{/stages/pool/json,null,AVAILABLE}

2017-02-08 11:18:44 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@74eab077{/storage,null,AVAILABLE}

2017-02-08 11:18:44 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@3063be68{/storage/json,null,AVAILABLE}

2017-02-08 11:18:44 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@3a3bc0da{/storage/rdd,null,AVAILABLE}

2017-02-08 11:18:44 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@2d2f09a4{/storage/rdd/json,null,AVAILABLE}

2017-02-08 11:18:44 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@c677d7e{/environment,null,AVAILABLE}

2017-02-08 11:18:44 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@215a0264{/environment/json,null,AVAILABLE}

2017-02-08 11:18:44 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@45832b85{/executors,null,AVAILABLE}

2017-02-08 11:18:44 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@103478b8{/executors/json,null,AVAILABLE}

2017-02-08 11:18:44 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@270f7b4d{/executors/threadDump,null,AVAILABLE}

2017-02-08 11:18:44 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@56b704ea{/executors/threadDump/json,null,AVAILABLE}

2017-02-08 11:18:44 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@ab4d2ba{/static,null,AVAILABLE}

2017-02-08 11:18:44 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@39f68aec{/,null,AVAILABLE}

2017-02-08 11:18:44 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@65ff4b8c{/api,null,AVAILABLE}

2017-02-08 11:18:44 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@b81938d{/stages/stage/kill,null,AVAILABLE}

2017-02-08 11:18:44 INFO  ServerConnector:266 - Started ServerConnector@3d8bd881{HTTP/1.1}{0.0.0.0:4040}

2017-02-08 11:18:44 INFO  Server:379 - Started @18429ms

2017-02-08 11:18:44 INFO  Utils:54 - Successfully started service 'SparkUI' on port 4040.

2017-02-08 11:18:44 INFO  SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://192.168.1.66:4040

2017-02-08 11:18:44 INFO  Executor:54 - Starting executor ID driver on host localhost

2017-02-08 11:18:44 INFO  Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 49348.

Please let me know if you need further information from my side and thanks again.

Alessandro

On Wednesday, February 8, 2017 at 1:38:58 AM UTC-6, Jason Dai wrote:

Hi Alessandro,

To post to this group, send email to bigdl-u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigdl-user-group/a10ffe19-96f5-45aa-a610-47fa37f43d2a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "BigDL User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bigdl-user-gro...@googlegroups.com.
To post to this group, send email to bigdl-us...@googlegroups.com.

[Jason Dai]

Hi Alessandro,

I wonder if you can share the actual command you use to run the program? And for a quick check, can you do something like "vectorizedRdd.coalesce(1, true).count()" to check the speed of data loading? And maybe you can also check the Java GC log to see if there are a lot of GCs?

In addition, there are two potential issues from the log (which should only impact the training throughput though):

1) There is warning of "Invalid env setting. Please use bigdl.sh set the env." Did you use BigDL to launch your program? See https://github.com/intel-analytics/BigDL/wiki/Getting-Started#spark-program-local-mode-and-cluster-mode for how to launch BigDL programs.

2) It seems you are using 8 cores on your local machine; it is recommended to set the number of cores to the number of physical cores (not logical processors) on the computer.

Thanks,

-Jason

To unsubscribe from this group and stop receiving emails from it, send an email to bigdl-user-group+unsubscribe@googlegroups.com.
To post to this group, send email to bigdl-user-group@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bigdl-user-group/94cfcf05-bd59-45ed-98bc-453056497b8a%40googlegroups.com.

[alepb...@gmail.com]

Jason,

The way I start my program is through the scala shell passing the compiled .jar"

scala -cp /dl/target/bigdl-0.1.0-SNAPSHOT-jar-with-dependencies-and-spark.jar -J-Xmx16g

For some reason I get different errors when I try to use an IDE (intelliJ) to work with BigDL. The downside is also the fact (as I mentioned in my previous message) that Im not able to monitor the SparkUI. Maybe you can tell me more reading this:

2017-02-07 10:12:44 WARN  Utils:66 - Your hostname, MACHINE_ID resolves to a loopback address: 127.0.0.1; using 10.36.XX.XXX instead (on interface en0)

2017-02-07 10:12:44 WARN  Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address

2017-02-07 10:12:44 INFO  SecurityManager:54 - Changing view acls to: MY_USER

2017-02-07 10:12:44 INFO  SecurityManager:54 - Changing modify acls to: MY_USER

2017-02-07 10:12:44 INFO  SecurityManager:54 - Changing view acls groups to: 

2017-02-07 10:12:44 INFO  SecurityManager:54 - Changing modify acls groups to: 

2017-02-07 10:12:44 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(MY_USER); groups with view permissions: Set(); users  with modify permissions: Set(MY_USER); groups with modify permissions: Set()

To answer your questions: when I run vectorizedRdd.coalesce(1, true) this is what I get:

scala> vectorizedRdd.coalesce(1, true).count()

2017-02-09 10:36:30 INFO  SparkContext:54 - Starting job: count at <console>:61

2017-02-09 11:01:44 INFO  DAGScheduler:54 - Job 5 finished: count at <console>:61, took 1514.288633 s

res0: Long = 257291

The problem is indeed the loading of the data as you can tell from the fact that it takes almost 25 minutes to load 25K lines. Please note that in this last run I changed the number of cores to 1 to follow your previous suggestion.

And this is a GC log snapshot. Hopefully you will find it useful for gaining more insights:

The heap seems to act weird but Im not sure Im able to do anything about it. 

Hopefully you will find these information useful to guide me further but thanks so far.

Alessa

...

[zhichao]

For the UI part, I think it's becuase the service would not be enable by default for local mode in Spark.  

Thanks,

Zhichao

--

You received this message because you are subscribed to the Google Groups "BigDL User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigdl-user-group+unsubscribe@googlegroups.com.
To post to this group, send email to bigdl-user-group@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigdl-user-group/9333f417-2d63-4ea4-a99f-329e9ac5df34%40googlegroups.com.

[zhichao]

8080 service would not be enable in local mode. but for the 4040 service you should see something like on the log:

 org.apache.spark.util.Utils main:58 Successfully started service 'SparkUI' on port 4040

and for the warning:

 2017-02-07 10:12:44 WARN  Utils:66 - Your hostname, MACHINE_ID resolves to a loopback address: 127.0.0.1; using 10.36.XX.XXX instead (on interface en0)

2017-02-07 10:12:44 WARN  Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address

I think it should be fine for using loopback address, but you can set the SPARK_LOCAL_IP="<IP address>" in spark-env.sh file or change the ip binding in /etc/hosts to eliminate that.

On Fri, Feb 10, 2017 at 1:44 AM, <alepb...@gmail.com> wrote:

--

You received this message because you are subscribed to the Google Groups "BigDL User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigdl-user-group+unsubscribe@googlegroups.com.
To post to this group, send email to bigdl-user-group@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigdl-user-group/9333f417-2d63-4ea4-a99f-329e9ac5df34%40googlegroups.com.

[zhichao]

Hi Alessandro,

Just tried on my local machine, it only took seconds to finish.

scala> vectorizedRdd.coalesce(1, true).count()

I think this is more about how spark loading data instead of BigDL. 

Anyway, the job UI might be able to give some hint on this, so could you elaborate more on "cannot access 'SparkUI'"? There might be some log indicate the reason why your http request fail each time you refresh the page.

Thanks,

Zhichao

[zhichao]

FYI.

I run it under Intellij with the default value.

25K mock records.

Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz.

1G memory(just for testing the data loading)

Spark_2.0 scala_2.11

[alepb...@gmail.com]

Zhichao,

I think what is still unclear to me is where/how can I change the spark env variables. Please note that so far I ve been starting BigDL passing the compiled jar to the scala shell, more precisely:

scala -cp /dl/target/bigdl-0.1.0-SNAPSHOT-jar-with-dependencies-and-spark.jar -J-Xmx16g

Because I compiled bigDl via:

mvn clean package -DskipTests -P mac -P spark_2.0

when I initialize my spark context I'm getting the following output:

scala> val sc = new SparkContext(Engine.init(1, 1, true).get.setMaster("local[*]").setAppName("W2V").set("spark.task.maxFailures", "1"))

2017-02-10 10:45:41 INFO  ThreadPool$:87 - Set mkl threads to 1 on thread 1

2017-02-10 10:45:41 WARN  Engine$:344 - Invalid env setting. Please use bigdl.sh set the env. For spark application, please use Engine.sparkConf() to initialize your sparkConf

2017-02-10 10:45:41 INFO  ThreadPool$:87 - Set mkl threads to 1 on thread 1

2017-02-10 10:45:41 WARN  SparkConf:66 - The configuration key 'spark.akka.frameSize' has been deprecated as of Spark 1.6 and may be removed in the future. Please use the new key 'spark.rpc.message.maxSize' instead.

2017-02-10 10:45:41 INFO  SparkContext:54 - Running Spark version 2.0.0

2017-02-10 10:45:41 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

2017-02-10 10:45:42 WARN  Utils:66 - Your hostname, MY_MACHINE_ID resolves to a loopback address: 127.0.0.1; using 10.36.XX.XXX instead (on interface en0)

2017-02-10 10:45:42 WARN  Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address

2017-02-10 10:45:42 INFO  SecurityManager:54 - Changing view acls to: MY_USER_ID

2017-02-10 10:45:42 INFO  SecurityManager:54 - Changing modify acls to: MY_USER_ID

2017-02-10 10:45:42 INFO  SecurityManager:54 - Changing view acls groups to: 

2017-02-10 10:45:42 INFO  SecurityManager:54 - Changing modify acls groups to: 

2017-02-10 10:45:42 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(MY_USER_ID); groups with view permissions: Set(); users  with modify permissions: Set(MY_USER_ID); groups with modify permissions: Set()

2017-02-10 10:45:42 INFO  Utils:54 - Successfully started service 'sparkDriver' on port 64233.

2017-02-10 10:45:42 INFO  SparkEnv:54 - Registering MapOutputTracker

2017-02-10 10:45:42 INFO  SparkEnv:54 - Registering BlockManagerMaster

2017-02-10 10:45:42 INFO  DiskBlockManager:54 - Created local directory at /private/var/folders/dy/13g7lrsj31d34ccjfm33hqcsqr4dwy/T/blockmgr-52ce438e-3b24-4403-834c-400184117cfb

2017-02-10 10:45:42 INFO  MemoryStore:54 - MemoryStore started with capacity 8.4 GB

2017-02-10 10:45:42 INFO  SparkEnv:54 - Registering OutputCommitCoordinator

2017-02-10 10:45:42 INFO  log:186 - Logging initialized @20908ms

2017-02-10 10:45:42 INFO  Server:327 - jetty-9.2.z-SNAPSHOT

2017-02-10 10:45:42 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@6722db6e{/jobs,null,AVAILABLE}

2017-02-10 10:45:42 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@18f20260{/jobs/json,null,AVAILABLE}

2017-02-10 10:45:42 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@4ae33a11{/jobs/job,null,AVAILABLE}

2017-02-10 10:45:42 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@7a48e6e2{/jobs/job/json,null,AVAILABLE}

2017-02-10 10:45:42 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@b40bb6e{/stages,null,AVAILABLE}

2017-02-10 10:45:42 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@3a94964{/stages/json,null,AVAILABLE}

2017-02-10 10:45:42 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@5049d8b2{/stages/stage,null,AVAILABLE}

2017-02-10 10:45:42 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@6d0b5baf{/stages/stage/json,null,AVAILABLE}

2017-02-10 10:45:42 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@631e06ab{/stages/pool,null,AVAILABLE}

2017-02-10 10:45:42 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@2a3591c5{/stages/pool/json,null,AVAILABLE}

2017-02-10 10:45:42 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@34a75079{/storage,null,AVAILABLE}

2017-02-10 10:45:42 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@346a361{/storage/json,null,AVAILABLE}

2017-02-10 10:45:42 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@107ed6fc{/storage/rdd,null,AVAILABLE}

2017-02-10 10:45:42 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@1643d68f{/storage/rdd/json,null,AVAILABLE}

2017-02-10 10:45:42 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@186978a6{/environment,null,AVAILABLE}

2017-02-10 10:45:42 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@2e029d61{/environment/json,null,AVAILABLE}

2017-02-10 10:45:42 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@482d776b{/executors,null,AVAILABLE}

2017-02-10 10:45:42 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@4052274f{/executors/json,null,AVAILABLE}

2017-02-10 10:45:42 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@132ddbab{/executors/threadDump,null,AVAILABLE}

2017-02-10 10:45:42 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@297ea53a{/executors/threadDump/json,null,AVAILABLE}

2017-02-10 10:45:42 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@acb0951{/static,null,AVAILABLE}

2017-02-10 10:45:42 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@5bf22f18{/,null,AVAILABLE}

2017-02-10 10:45:42 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@267f474e{/api,null,AVAILABLE}

2017-02-10 10:45:42 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@7a7471ce{/stages/stage/kill,null,AVAILABLE}

2017-02-10 10:45:42 INFO  ServerConnector:266 - Started ServerConnector@3f2ef586{HTTP/1.1}{0.0.0.0:4040}

2017-02-10 10:45:42 INFO  Server:379 - Started @21002ms

2017-02-10 10:45:42 INFO  Utils:54 - Successfully started service 'SparkUI' on port 4040.

2017-02-10 10:45:42 INFO  SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://10.36.XX.XXX:4040

2017-02-10 10:45:42 INFO  Executor:54 - Starting executor ID driver on host localhost

2017-02-10 10:45:42 INFO  Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 64234.

2017-02-10 10:45:42 INFO  NettyBlockTransferService:54 - Server created on 10.36.XX.XXX:64234

2017-02-10 10:45:42 INFO  BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver,10.36.XX.XXX, 64234)

2017-02-10 10:45:42 INFO  BlockManagerMasterEndpoint:54 - Registering block manager 10.36.XX.XXX:64234 with 8.4 GB RAM, BlockManagerId(driver, 10.36.XX.XXX, 64234)

2017-02-10 10:45:42 INFO  BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, 10.36.XX.XXX, 64234)

2017-02-10 10:45:42 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@6b739528{/metrics/json,null,AVAILABLE}

sc: org.apache.spark.SparkContext = org.apache.spark.SparkContext@3be8821f

It looks like the ui succesfully starts the UI from the log:

2017-02-10 10:45:42 INFO  SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://10.36.XX.XXX:4040

But in reality, when I try to access via browser to http://10.36.XX.XXX:4040  I get the following error:

scala> 2017-02-10 10:39:45 WARN  HttpChannel:384 - /

java.lang.NoSuchMethodError: javax.servlet.http.HttpServletRequest.isAsyncStarted()Z

at org.spark_project.jetty.servlets.gzip.GzipHandler.handle(GzipHandler.java:484)

at org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)

at org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

at org.spark_project.jetty.server.Server.handle(Server.java:499)

at org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:311)

at org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)

at org.spark_project.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)

at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)

at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)

at java.lang.Thread.run(Thread.java:745)

2017-02-10 10:39:45 WARN  HttpChannel:482 - Could not send response error 500: java.lang.NoSuchMethodError: javax.servlet.http.HttpServletRequest.isAsyncStarted()Z

2017-02-10 10:39:45 WARN  HttpChannel:384 - /jobs/

java.lang.NoSuchMethodError: javax.servlet.http.HttpServletRequest.isAsyncStarted()Z

at org.spark_project.jetty.servlets.gzip.GzipHandler.handle(GzipHandler.java:484)

at org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)

at org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

at org.spark_project.jetty.server.Server.handle(Server.java:499)

at org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:311)

at org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)

at org.spark_project.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)

at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)

at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)

at java.lang.Thread.run(Thread.java:745)

Hopefully this gives you some more context to help me fix the issue. 

Ps. Im using scala 2.11.7 

Thanks,

Alessandro

...

[alepb...@gmail.com]

Although my questions from the previous message remain I was able to figure out the main problem with the loading of my data.

It has nothing to do with Spark, rather with Scala itself. Specifically with Scala 2.11.

I have a function that I use to generate my vectors from my w2vec model. The same function calls, among others, the method scala.collection.IndexedSeqOptimized$class.slice() which seems to be incredibly slow. 

Digging on the web I found this:

http://stackoverflow.com/questions/37969193/why-is-array-slice-so-shockingly-slow

which basically highlights a bug in scala 2.11 although fixed in 2.12.

Unfortunately if I use scala 2.12 BigDL seems to fail so I dont have other option but changing my function in a way that method doesnt get called. 

I just wanted to share this even though it might not be that helpful for other. 

Id still be interested in a feedback to my previous message. Thanks!

Alessandro

[zhichao]

For BigDL, it require some environment variables via "source PATH_To_BigDL/scripts/bigdl.sh" before starting. Pls refer to: https://github.com/intel-analytics/BigDL/wiki/Getting-Started.

For the UI exception you provided, it's due to BigDL included multiple version of servlet-api. We would create a patch for this(https://github.com/intel-analytics/BigDL/issues/458) and I think you can remove javax.servlet:servlet-api:2.5 for now to remedy the problem.

Thanks,

Zhichao

--

You received this message because you are subscribed to the Google Groups "BigDL User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigdl-user-group+unsubscribe@googlegroups.com.
To post to this group, send email to bigdl-user-group@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigdl-user-group/dbc218d7-6101-44c0-a81c-50fd6ad0d4ae%40googlegroups.com.

[alepb...@gmail.com]

Zhichao,

how would you remove  javax.servlet:servlet-api:2.5 temporarily? Should I re-compile BigDl editing the pom.xml?

Thanks

[zhichao]

You can remove it from Intellij directly from Project Structure/Libraries, or modify the pom.xml.
I've just fix it in https://github.com/intel-analytics/BigDL/issues/458.
Let me know if you still facing the problem for the latest code.

Thanks

--

You received this message because you are subscribed to the Google Groups "BigDL User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigdl-user-group+unsubscribe@googlegroups.com.
To post to this group, send email to bigdl-user-group@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigdl-user-group/d8cf270f-5a62-49e5-bbb6-c18aa6d042cc%40googlegroups.com.

[alepb...@gmail.com]

I compiled the latest BigDL.

mvn clean package -DskipTests -P mac -P spark_2.0

Execute:

source scripts/bigdl.sh

And use the Scala Shell(I cant use IntelliJ because I get memory problems which Im not sure they would be easy to solve). So:

scala -cp $BIGDL_HOME/dl/target/bigdl-0.1.0-SNAPSHOT-jar-with-dependencies-and-spark.jar -J-Xmx16g 

Start the spark context :

2017-02-14 12:40:38 INFO  Server:379 - Started @23817ms

2017-02-14 12:40:38 INFO  Utils:54 - Successfully started service 'SparkUI' on port 4041.

2017-02-14 12:40:38 INFO  SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://10.36.XX.XXX:4041

2017-02-14 12:40:38 INFO  Executor:54 - Starting executor ID driver on host localhost

2017-02-14 12:40:38 INFO  Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 58878.

2017-02-14 12:40:38 INFO  NettyBlockTransferService:54 - Server created on 10.36.XX.XXX:58878

2017-02-14 12:40:38 INFO  BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver,10.36.XX.XXX, 58878)

2017-02-14 12:40:38 INFO  BlockManagerMasterEndpoint:54 - Registering block manager 10.36.XX.XXX:58878 with 8.4 GB RAM, BlockManagerId(driver, 10.36.XX.XXX, 58878)

2017-02-14 12:40:38 INFO  BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, 10.36.XX.XXX, 58878)

2017-02-14 12:40:38 INFO  ContextHandler:744 - Started o.s.j.s.ServletContextHandler@7bc2ae16{/metrics/json,null,AVAILABLE}

but as soon as I try to access the UI from the browser I get this:

scala> 2017-02-14 12:40:50 WARN  HttpChannel:384 - /

java.lang.NoSuchMethodError: javax.servlet.http.HttpServletRequest.isAsyncStarted()Z

at org.spark_project.jetty.servlets.gzip.GzipHandler.handle(GzipHandler.java:484)

at org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)

at org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

at org.spark_project.jetty.server.Server.handle(Server.java:499)

at org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:311)

at org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)

at org.spark_project.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)

at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)

at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)

at java.lang.Thread.run(Thread.java:745)

2017-02-14 12:40:50 WARN  HttpChannel:482 - Could not send response error 500: java.lang.NoSuchMethodError: javax.servlet.http.HttpServletRequest.isAsyncStarted()Z

2017-02-14 12:40:50 WARN  HttpChannel:384 - /jobs/

java.lang.NoSuchMethodError: javax.servlet.http.HttpServletRequest.isAsyncStarted()Z

at org.spark_project.jetty.servlets.gzip.GzipHandler.handle(GzipHandler.java:484)

at org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)

at org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

at org.spark_project.jetty.server.Server.handle(Server.java:499)

at org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:311)

at org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)

at org.spark_project.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)

at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)

at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)

at java.lang.Thread.run(Thread.java:745)

2017-02-14 12:40:50 WARN  QueuedThreadPool:610 - 

java.lang.NoSuchMethodError: javax.servlet.http.HttpServletResponse.getStatus()I

at org.spark_project.jetty.server.handler.ErrorHandler.handle(ErrorHandler.java:112)

at org.spark_project.jetty.server.Response.sendError(Response.java:597)

at org.spark_project.jetty.server.HttpChannel.handleException(HttpChannel.java:487)

at org.spark_project.jetty.server.HttpConnection$HttpChannelOverHttp.handleException(HttpConnection.java:594)

at org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:387)

at org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)

at org.spark_project.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)

at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)

at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)

at java.lang.Thread.run(Thread.java:745)

2017-02-14 12:40:50 WARN  QueuedThreadPool:617 - Unexpected thread death: org.spark_project.jetty.util.thread.QueuedThreadPool$3@639f749f in SparkUI{STARTED,8<=8<=200,i=2,q=0}

Not sure what else I should do...

To unsubscribe from this group and stop receiving emails from it, send an email to bigdl-user-gro...@googlegroups.com.
To post to this group, send email to bigdl-us...@googlegroups.com.

[zhichao]

Hi Alessandro,

I tested on Ubuntu, might try Mac later.

In the mean while, if you want to run on Spark in interactive way, it would be more formal to execute like this:

source scripts/bigdl.sh

$SPARK_HOME/bin/spark-shell --jars ~/bin/god/BigDL/dist/lib/bigdl-0.1.0-SNAPSHOT-jar-with-dependencies.jar

Thanks,

Zhichao

To unsubscribe from this group and stop receiving emails from it, send an email to bigdl-user-group+unsubscribe@googlegroups.com.
To post to this group, send email to bigdl-user-group@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bigdl-user-group/7a691c83-656d-4c7c-b502-92b87026551d%40googlegroups.com.

[alepb...@gmail.com]

Zhichao,

using spark-shell + bigDL jar solved the UI issue. I am now able to monitor my application. Thanks for the support.

val sc = new SparkContext(Engine.init(nodeNum, coreNum, true).get.setMaster("local[*]").setAppName("test").set("spark.task.maxFailures", "1&