Custom Metric in H2O

[Lorenzo Isella]

Dear All,
I use h2o from R. One question: how can I define a custom metric
(e.g. the mean absolute error instead of the mean square error) for a
regression problem?
Many Thanks

Lorenzo

[Lauren DiPerna]

MAE exists in the current version of H2O as `h2o.mae()`, please update your version if it doesn't have it.

example:

library(h2o)

h <- h2o.init()

fr <- as.h2o(iris)

m <- h2o.deeplearning(x=2:5,y=1,training_frame=fr)

h2o.mae(m)

cheers,

Lauren

Lorenzo

--
You received this message because you are subscribed to the Google Groups "H2O Open Source Scalable Machine Learning  - h2ostream" group.
To unsubscribe from this group and stop receiving emails from it, send an email to h2ostream+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Lorenzo Isella]

Thanks! I was acrually running an obsolete version of h20.
However, suppose that I need a metric which is not available of the
shelf (e.g. log(mean absolute error+1)).
How can I implement that?
Many thanks

Lorenzo

On Mon, Oct 31, 2016 at 08:07:00PM -0700, Lauren DiPerna wrote:
>MAE exists in the current version of H2O as `h2o.mae()`, please update your
>version if it doesn't have it.
>
>
>example:
>
>library(h2o)
>h <- h2o.init()
>fr <- as.h2o(iris)
>m <- h2o.deeplearning(x=2:5,y=1,training_frame=fr)
>h2o.mae(m)
>
>cheers,
>
>Lauren
>
>On Mon, Oct 31, 2016 at 7:40 AM, Lorenzo Isella <lorenzo...@gmail.com>
>wrote:
>
>> Dear All,
>> I use h2o from R. One question: how can I define a custom metric
>> (e.g. the mean absolute error instead of the mean square error) for a
>> regression problem?
>> Many Thanks
>>
>> Lorenzo
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "H2O Open Source Scalable Machine Learning - h2ostream" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to h2ostream+...@googlegroups.com.

[Darren Cook]

> Thanks! I was acrually running an obsolete version of h20.
> However, suppose that I need a metric which is not available of the
> shelf (e.g. log(mean absolute error+1)).
> How can I implement that?

The return value of `h2o.predict()` gives you its prediction for each
sample. So you can do calculations directly on them. I think you can
even do most operations without downloading into R. E.g.

test = h2o.getFrame("test")
p = h2o.predict(m, test) #Where m is your model
tmp = h2o.cbind(test[,"answer"], p)
myMetric = mean( log( abs(tmp[,1] - tmp[,2]) + 1) )

Untested, but I think that gives the mean of log-absolute-error-plus-one!

(Substitute `valid` for `test`, to get the metric on your validation
data set, etc.)

Darren



--
Darren Cook, Software Researcher/Developer
My New Book: Practical Machine Learning with H2O,
published by O'Reilly. If interested, let me know and
I'll send you a discount code as soon it is released.

[Lorenzo Isella]

Apologies, I understand I did not make myself clear.
My question is how I can use a custom metric as a stopping metric.
E.g. how I can use mae or something else as a stopping metric.
Cheers

Lorenzo

#####################################

Thanks! I was acrually running an obsolete version of h20.
However, suppose that I need a metric which is not available of the
shelf (e.g. log(mean absolute error+1)).
How can I implement that?

Many thanks

Lorenzo


On Mon, Oct 31, 2016 at 08:07:00PM -0700, Lauren DiPerna wrote:

>MAE exists in the current version of H2O as `h2o.mae()`, please update your
>version if it doesn't have it.
>
>
>example:
>
>library(h2o)
>h <- h2o.init()
>fr <- as.h2o(iris)
>m <- h2o.deeplearning(x=2:5,y=1,training_frame=fr)
>h2o.mae(m)
>
>cheers,
>
>Lauren
>
>On Mon, Oct 31, 2016 at 7:40 AM, Lorenzo Isella <lorenzo...@gmail.com>
>wrote:
>
>> Dear All,
>> I use h2o from R. One question: how can I define a custom metric
>> (e.g. the mean absolute error instead of the mean square error) for a
>> regression problem?
>> Many Thanks
>>
>> Lorenzo
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "H2O Open Source Scalable Machine Learning - h2ostream" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to h2ostream+...@googlegroups.com.

[Lauren DiPerna]

if you want to add a custom metric to your own build, or make a PR request you can follow the steps taken to create the rmse, for example.

here is the commit, poke around to learn where you need to make your updates:

https://github.com/h2oai/h2o-3/commit/666184f27b84d4b7e5165e881b8bf05f1fd88838

email to h2ostream+unsubscribe@googlegroups.com.

[kaseyr...@gmail.com]

I am not following, but I have the same question. I want to set stopping_metric = "mae" or my own custom function. But it will not let me.
Is this possible?

[Darren Cook]

> I am not following, but I have the same question. I want to set
> stopping_metric = "mae" or my own custom function. But it will not
> let me. Is this possible?

1. To use MAE, make sure you use a recent version (it appeared 2-3
months ago). (The error message should be telling you which metrics are
available?)

2. To use your own custom function you will need to add a new metric to
the H2O *Java* source code and compile it in. Lauren's reply was giving
a starting point for that.
(Or pay someone to do it, or put in a feature request in
https://0xdata.atlassian.net/projects/PUBDEV/summary and hope lots of
people have the same need and it eventually gets done.)