I cannot load my model >>> FileNotFoundError: [Errno 2] No such file or directory

[valer...@gmail.com]

Here you can find a Jupyter Notebook that shows the problem:

https://github.com/PHJT003/sos/blob/main/hddm/error-loading-model.ipynb

Dear users,

I run an HDDM on the cluster of my University and saved it with the hddm.save() function. I passed to this function a string in the form of "directory/mymodel".

I checked on the cluster and everything was there as I expected: I successfully fitted the model and saved it in its destination folder (i.e., the "directory" in bold above), exported my CSV files, etc. All the outputs seemed fine.

After this, I moved all the files to a local machine... I wanted to continue my analyses on a Jupyter Notebook because the cluster only works via terminal and I can't plot the posteriors there.

Unfortunately, I couldn't load my model and got this error:

FileNotFoundError: [Errno 2] No such file or directory: '/MRIWork/MRIWork08/mt/valerio_villani/hddm-weapon-kobe/cardiac-wit/data/saved-models/model__HDDMRegr_heart-2-params_m04.db'

I believe this has something to do with relative paths and the fact that I saved my model with hddm.save("directory/mymodel").

In light of this, is there a way for me to load my model with hddm.load() or the pickle library at all?

The model took 8 days to fit and I hope I could use the files I saved instead of re-running it from scratch...

Thank you very much for your time and attention, I look forward to hearing from you.

All the best,

Valerio.

[valer...@gmail.com]

When I load the model on my PC, the error message shows the absolute path of the cluster.

Cluster path

/MRIWork/MRIWork08/mt/valerio_villani/hddm-weapon-kobe/cardiac-wit/data/saved-models/model__HDDMRegr_heart-2-params_m04.db

On my PC, the path where I have the saved model is slightly different, of course.

My PC path

/home/valerio/hddm/hddm-weapon-kobe/cardiac-wit/data/saved-models/model__HDDMRegr_heart-2-params_m04.db

However, I use relative paths (pyprojroot library) in my code so that the bold text can vary dynamically, depending on the machine I use.

It looks like hddm.load() is looking for the file in the "cluster path", even if I'm providing "my PC path".

[valer...@gmail.com]

Hello everybody,

I've created the path where hddm.load() looks for the model and put the files of interest there.

However, I am getting another error now:

AttributeError: 'DataFrame' object has no attribute '_data'

Below you can find the Jupyter Notebook:

https://github.com/PHJT003/sos/blob/main/hddm/no-attribute-data.ipynb

Does anybody have an idea on how to fix this?

Thanks a lot,

Valerio

[valer...@gmail.com]

Hello, community...

Despite many attempts, I still have issues when loading models.

I made sure I save my HDDMs with the basic syntax model.save("mymodel"), to avoid embedding an absolute path into my pickled database (cf. 1st message of this thread).

However, when I run model.load(), I still get the same error (i.e., AttributeError: 'DataFrame' object has no attribute '_data'):

https://github.com/PHJT003/sos/blob/main/hddm/error-loading-model.ipynb

Does anyone know why this is happening?

How can I load the model?

Thanks, I look forward to your reply.

Best wishes,

Valerio

P.S.: I deleted the notebook https://github.com/PHJT003/sos/blob/main/hddm/no-attribute-data.ipynb as it was outdated.

[Zhamilya Gazman]

Hi Valerio,

I had a similar issue. I fixed it by working in the same folder where my model has been saved (setting the current working directory to the folder where I saved my model). 

Hope this helps,

Z

[Anne Urai]

Hi Valerio,

another approach is to save only the traces and point estimates to csv files, which you can then easily copy to your local computer:

see https://github.com/anne-urai/2019_Urai_choice-history-ddm/blob/1a2f168ca05ff9759cc3ede4b5ead7f22df098ae/fitHDDM.py#L182

--
You received this message because you are subscribed to the Google Groups "hddm-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hddm-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hddm-users/f617e211-25a0-472b-bddc-0d1ed4874127n%40googlegroups.com.

[valer...@gmail.com]

Thank you all for the useful advice, I will keep this in mind.

Eventually, I fixed the error, but I am not sure what did the trick.

For the record, these are some of the things that I tried on my Linux machine:

• restart everything;
• run sudo apt-get update -y && sudo apt-get upgrade -y;
  
• up- and down-grade Pandas (cf. here);
• re-install Jupyter Notebook;
• install Jupyter Lab;
• (re-)install the IPython Kernel for my Anaconda environment with the HDDM package;
• copy my Anaconda environment;
• delete and re-install my Anaconda environment from scratch.

I suspect that something might have broken in my virtual environment.

Thus, this might have been the winning combination of actions: create the Anaconda environment from scratch + install Jupyter Notebook and the kernel + use an older version of Pandas.

Best,

V

[hcp...@gmail.com]

Hi, Valerio,

Great to hear that you solved this issue, I can imagine the time you spent on this.

Also, thanks a lot for sharing your experience. Would you mind also share the version information that works? e.g., the version of linux, pandas, jupyter? This information might be helpful for future users ;)

Best,

Chuan-Peng

[valer...@gmail.com]

Hi Chuan-Peng,

I should have done this before really, thanks for nudging me! :)

I use the HDDM package on a Linux OS.

Distributor ID: Ubuntu

Description: Ubuntu 20.04.3 LTS

Release: 20.04

Codename: focal

The packages below are part of an Anaconda virtual environment (conda 4.10.3) that was created from a YML file Chuan-Peng exported.

hddm             : 0.8.0

jupyter-notebook : 6.0.3

pandas           : 1.0.4

If you want to add the IPython Kernel to the Jupyter Notebook, use the following syntax in your terminal.

conda activate hddm_env

python -m ipykernel install --user --name hddm_env --display-name "HDDM"

where hddm_env is your Anaconda environment where you installed the HDDM package.

If you do not do this bit, you might get an error when importing the HDDM package (and all the dependencies installed in your environment).

All the best,

Valerio