Unable to unpickle a pickled h2o model object

[Changyao Chen]

Hi there

I am using the h2o python module on Hadoop (Hortonworks hdp 2.4.2, h2o cluster version 3.18.0.8, Python 3.6.1). I construct a simple h2o model as:

_clf = H2OGradientBoostingEstimator({'balance_classes': True, 'distribution': 'bernoulli', 'ntrees':10, 'max_depth':3})

I then use python's pickle module to pickle the model object:

import pickle
model = pickle.dump(_clf, open(filename, 'wb'), protocol=2)

However, when I tried to unpickle the model with the following code:

loaded_model = pickle.load(open(filename, 'rb'))

it returns the following error:

TypeError: __new__() missing 1 required positional argument: 'keyvals'

If I insist to use python's pickle module (instead of h2o's own h2o.save_model(), but this is not a hard requirement), is there a quick fix?

Thanks!

[Changyao Chen]

One thing I forgot to mention is that, after the h2o model instantiation, I also trained the model, with the following code: 

_clf.train(x=train_X.columns, y=config.label_col[0], training_frame=train_frame, validation_frame=valid_frame)

Not sure whether it matters, but just for completeness..

Thanks!

[Darren Cook]

> I then use python's pickle module to pickle the model object:

> ...

> If I insist to use python's pickle module (instead of h2o's own
> h2o.save_model(), but this is not a hard requirement), is there a quick fix?

The model (your `_clf`) is just a bunch of meta data about the model.
The real model exists on your h2o cluster.
So, use `h2o.save_model()`.

If you really want to use pickle, you will need to use
`h2o.save_model()` then load it in to python as a binary object, and
then pickle that. And a similar process for unpickling.

Darren



--
Darren Cook, Software Researcher/Developer
My New Book: Practical Machine Learning with H2O:
http://shop.oreilly.com/product/0636920053170.do

[xyy6...@gmail.com]

Could I save other h2o objects such as "h2o.model.metrics_base.H2OBinomialModelMetrics"?

In R, if using other ML package, it is easy to save any object one or in batch to binary file(s) by calling

save()

saveRDS()

save.image()

I have a hard time using any function, including json.dump, pickle methods and h2o.save_model(), to deal with h2o non-model object....

I appreciate if anyone could give a solution.

Thanks!!

[Yu Cao]

Hi Darren:

I know it's been a long time but I am having exactly the same issue. Could you be more specific on your solution, please?

I saved my model using h2o.saveModel() in R and load it in python using h2o.load_model(); however, I got the same error message when trying to pickle it.

Thank you.

[Darren Cook]

On 28/09/2023 21:32, Yu Cao wrote:
> Hi Darren:
>
> I know it's been a long time but I am having exactly the same issue. Could
> you be more specific on your solution, please?
>
> I saved my model using h2o.saveModel() in R and load it in python using
> h2o.load_model(); however, I got the same error message when trying to
> pickle it.

By "load it in to python as a binary object" I meant find the binary
file and load it directly from disk, using python's `open()` function,
not using any h2o function.

Just a reminder that, outside of some exotic use cases, if you are using
pickle with H2O then you have misunderstood how H2O works and are using
the wrong tool for the job. See also
https://docs.h2o.ai/h2o/latest-stable/h2o-docs/save-and-load-model.html

Darren