Optimise Deep Learning Model with R

[Julian Hillebrand]

I created a deep learning model with h2o in R. But it still has very poor performance. How can I optimise the outcome? What parameters can I how change? Is it just randomly changing parameters and see what happens to the outcome?

I already integrated grid search and automatic parameter tuning. How should I change the hidden layers? Is there a concept to further optimise the number of layers?

This is my code so far:

res.dl <- h2o.deeplearning(x = 2:23, y = 1, data = trData,classification=TRUE, 

activation = "Tanh",hidden=rep(160,150,200),epochs = 20)

pred_labels <- h2o.predict(res.dl, tsData)[,1]
actual_labels <- tsData[,1]
cm <- h2o.confusionMatrix(pred_labels,actual_labels)

res.dl@model$confusion

#grid search
grid_search <- h2o.deeplearning(x=c(2:23), y=1, data=trData, ,classification=TRUE,validation=tsData,
                                 hidden=list(c(10,10),c(20,20)), epochs=0.1,
                                 activation=c("Tanh", "Rectifier"), l1=c(0,1e-5))
best_model <- grid_search@model[[1]]
best_model
best_params <- best_model@model$params
    best_params$activation
best_params$hidden
    best_params$l1


#Hyper-parameter Tuning with Random Search

models <- c()
for (i in 1:10) {
  rand_activation <- c("TanhWithDropout", "RectifierWithDropout")[sample(1:2,1)]
  rand_numlayers <- sample(2:5,1)
  rand_hidden <- c(sample(10:50,rand_numlayers,T))
  rand_l1 <- runif(1, 0, 1e-3)
  rand_l2 <- runif(1, 0, 1e-3)
  rand_dropout <- c(runif(rand_numlayers, 0, 0.6))
  rand_input_dropout <- runif(1, 0, 0.5)
  dlmodel <- h2o.deeplearning(x=2:23, y=1, data=trData, validation=tsData, ,classification=TRUE,epochs=0.1,
                              activation=rand_activation, hidden=rand_hidden, l1=rand_l1, l2=rand_l2,
                              input_dropout_ratio=rand_input_dropout, hidden_dropout_ratios=rand_dropout)
  models <- c(models, dlmodel)
}


best_err <- best_model@model$valid_class_error #best model from grid search above


for (i in 1:length(models)) {
  err <- models[[i]]@model$valid_class_error
      if (err < best_err) {
        best_err <- err
        best_model <- models[[i]]
      }
    }
    best_model
    best_params <- best_model@model$params
best_params$activation
    best_params$hidden
best_params$l1
    best_params$l2
best_params$input_dropout_ratio
    best_params$hidden_dropout_ratios




dlmodel_continued <- h2o.deeplearning(x=c(2:23), y=1, data=trData, validation=tsData,,classification=TRUE,
                                       checkpoint = best_model, l1=best_params$l1, l2=best_params$l2, epochs=0.5)

dlmodel_continued@model$valid_class_error


#train further
dlmodel_continued_again <- h2o.deeplearning(x=c(2:23), y=1, data=trData, validation=tsData,,classification=TRUE,
                                             checkpoint = dlmodel_continued, l1=best_params$l1, epochs=0.5)

dlmodel_continued_again@model$valid_class_error

Thank you in advance.

Regards

[Erin LeDell]

Julian,

First you want to try more epochs, say 400, and try a Rectifier activation function.  Next try different architectures ("hidden" param), say c(200, 200) or c(500, 500, 500).

But before spending a lot more time on tuning DL parameters, it would be a good idea to benchmark performance against another algorithm, such as Random Forest.  It could be that you have already squeezed all the predictive power out of your training data. 

If your DLs are not close to the performance of an RF, you should spend more time playing around with DL params.  If you are close to the performance of a RF, you may want to spend more time creating new features and take a break from tuning.

Another approach you can try is an ensemble of multiple learners: https://github.com/h2oai/h2o/tree/master/R/ensemble

-Erin

--
You received this message because you are subscribed to the Google Groups "H2O & Open Source Scalable Machine Learning - h2ostream" group.
To unsubscribe from this group and stop receiving emails from it, send an email to h2ostream+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Julian Hillebrand]

Thank you Erin.

The link with the ensemble example helped a lot.

I ended up with a AUC value of over 0.85.

The last part of my ensemble code looks like this:

metalearner <- c("SL.nnls")
family <- "binomial"
fit <- h2o.ensemble(x = 2:23, y = 1, data = trData, family = family, learner = learner, metalearner = metalearner, cvControl = list(V=20))


#Performance
pred <- predict(fit, tsData)
labels <- as.data.frame(tsData[,1])[,1]
# Ensemble test AUC
AUC(predictions=as.data.frame(pred$pred)[,1], labels=labels)

I used the example code to calculate the AUC value. But how can I now calculate other measures like the error rate or the accuracy? As the fit object is not

recognized as a h2o model.

And do you know what happens with the threshold? How can I see what threshold the model is using as it is a binary classifier.

Thank you.

Regards

[Erin LeDell]

Hi Julian,

Good to hear the ensemble is working for you.  I think in the future, the h2o.ensemble function will produce an object more similar to the objects produced by the other h2o base ML methods.  However, right now the model is stored as an R list, and the predict function generates predicted values only.  It does not do automatic thresholding.

The `preds` vector in the `h2o.ensemble` example is just the predicted values on the test set, so if you want to calculate other metrics like accuracy, you need to set a threshold and generate predicted labels first.  I think the default threshold in H2O is 0.5, but you can use whatever you want.

Here is an extension to the example code that will calculate the error rate with a threshold of 0.5:

predictions <- as.data.frame(pred$pred)[,1]  #pull preds into an R vector

predlabels <- ifelse(predictions > 0.5, 1, 0)  #above 0.5, set to label 1, else 0
error <- sum(!identical(predlabels, labels))/length(labels)  #error rate

Let me know if that gives you what you are looking for.

-Erin

[Erin LeDell]

Julian,

There is a typo in the previous code, the error should be calculated as follows:

error <- sum(!(predlabels==labels))/length(labels)

-Erin

[Julian Hillebrand]

This is exactly I was looking for. 

Thanks for your help and please keep up your great work!

Regards 

Julian

[hirosh...@gmail.com]

I had a very trouble in using the deep learning. actually even though I had a number of tries but still even I can not achieve the performance of RF. Especially, I am focusing on the time series forecast domain. If anyone can provide the some advices, it is appreciated.

Hiroshi Chin


2015年3月5日木曜日 10時33分40秒 UTC+9 Julian Hillebrand:

[hirosh...@gmail.com]