Lenskit.batch.predict persistance

[miguelga...@gmail.com]

Hello,

I am using the Lenskit.batch.predict method and I noticed that it persists the models in order to allow it to make a parallel batch prediction. However, I am currently running my evaluation script in a very limited machine regarding disk-space. I was wondering if there is any way of evaluating my models, using the LensKit library, without the need of saving the models in the temporary folder.

This the evaluating method I am currently using:

Thanks in advance.

Best regards,

Miguel

[Michael Ekstrand]

Hi Miguel,

There are two ways that you can reduce the disk space usage. One is to turn off parallel evaluation - if you specify n_jobs=1 to batch.predict, it will only use a single process and will not persist the model.

The other is to use Python 3.8 or newer. Python 3.8 provides robust shared memory facilities, and if they are available, LensKit will use them to persist the model into shared memory instead of a file on disk. This requires enough memory for two copies of the model, because it needs to be copied into shared memory. If you use train_isolated prior to predict, it will drop the original model object after copying, so all you have in memory is the one model. (Note that you have to call .close() on the result object from train_isolated after you are done predicting to free up the shared memory.)

- Michael

--
You received this message because you are subscribed to the Google Groups "LensKit Recommender Toolkit Development and Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lenskit-recsy...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lenskit-recsys/25c65d1a-8ac3-4eef-96e0-49d12125e519n%40googlegroups.com.

--

Michael D. Ekstrand — michael...@boisestate.edu — https://md.ekstrandom.net
Assistant Professor, Dept. of Computer Science, Boise State University

People and Information Research Team (PIReT) — http://coen.boisestate.edu/piret/

I may send mail outside of working hours; I do not expect you to. He/him.