How can I compare models?

[charle...@gmail.com]

Hello,

I new to h20 so please have mercy on me. I'm trying to compare two models (gpm and deeplearning) and if possible I would like to have a single metric (like F1) to do the comparison.

Below you will find my complete code for a gpm model. When I execute the last line:

h2o.F1(test_performance)

I get the error message:

Error in h2o.metric(object, thresholds, "f1") :
No f1 for H2OMultinomialMetrics
In addition: Warning message:
In h2o.performance(best_model, iris.train) :
Given data is same as the training data. Returning the training metrics.

Any suggestions on how I can compare two models will be be greatly appreciated.

Charles

This is what I have tried so far:

library(h2o)
localH2O <- h2o.init(ip = "localhost", port = 54321, startH2O = TRUE)
h2o.clusterInfo(localH2O)

irisPath = system.file("extdata", "iris.csv", package="h2o")
iris.hex = h2o.importFile(localH2O, path = irisPath)
summary(iris.hex)
head(iris.hex)

rnd <- h2o.runif(iris.hex)
iris.train <- h2o.assign(iris.hex[rnd < 0.7,], "iris.train")
iris.validate <- h2o.assign(iris.hex[rnd >= 0.7 & rnd < 0.9,], "iris.validate")
iris.test <- h2o.assign(iris.hex[rnd >= 0.9,], "iris.test")

models <- c()
for (loop in 1:10) {
rand_numtrees <- sample(1:50,1) ## 1 to 50 trees
rand_max_depth <- sample(5:15,1) ## 5 to 15 max depth
rand_min_rows <- sample(1:10,1) ## 1 to 10 min rows
rand_learn_rate <- 0.025*sample(1:10,1) ## 0.025 to 0.25 learning rate
model_name <- paste0("gbm_", loop,
"_ntrees", rand_numtrees,
"_maxdepth", rand_max_depth,
"_minrows", rand_min_rows,
"_learnrate", rand_learn_rate
)
model <- h2o.gbm(x = 1:4,
y = 5,
training_frame = iris.train,
validation_frame = iris.validate,
model_id = model_name,
distribution = "multinomial",
ntrees = rand_numtrees,
max_depth = rand_max_depth,
min_rows = rand_min_rows,
learn_rate = rand_learn_rate
)
models <- c(models, model)
}

## Find the best model (lowest logloss on the validation holdout set)
best_err <- 1e3
for (i in 1:length(models)) {
err <- h2o.logloss( h2o.performance(models[[i]], iris.validate) )
if (err < best_err) {
best_err <- err
best_model <- models[[i]]
}
}

## Show the "winning" parameters
best <- best_model@allparameters
best$ntrees
best$max_depth
best$min_rows
best$learn_rate

## Training set performance metrics
train_perf <- h2o.performance(best_model, iris.train)
h2o.confusionMatrix(train_perf)
h2o.logloss(train_perf)

## Validation set performance metrics
valid_perf <- h2o.performance(best_model, iris.validate)
h2o.confusionMatrix(valid_perf)
h2o.logloss(valid_perf)

## Build Final Model using the Full Training Data
iris.gbm = h2o.gbm(x = 1:4,
y = 5,
training_frame = iris.train,
activation = "Tanh",
validation_frame = iris.validate,
model_id = "final_model",
distribution = "multinomial",
ntrees = best$ntrees,
max_depth = best$max_depth,
min_rows = best$min_rows,
learn_rate = best$learn_rate
)

test_performance <- h2o.performance(best_model, iris.test)

h2o.logloss(test_performance)
h2o.confusionMatrix(test_performance)

predictions <- h2o.predict(iris.gbm, newdata=iris.test, measure = "F1")
print(predictions)

h2o.F1(test_performance)

[ccl...@gmail.com]

Hi! Metrics like F1 only apply to binomial models, ie 2 classes.
You trained on the iris dataset, which has THREE output classes.
At this point, I'm not the expert on comparing multinomial models, but you might look at the overall MSE, or max-error-per-class or similar metrics.

Also, the test/train warning was simply pointing out that your predictions are based on your training data - which can lead to overfitting, and unrealistically high performance scores for all models.

Cliff