Loading saved word2vec model

[Keshav Mathur]

Hello,

I recently trained a word2vec model on a large dataset using gensim while following this tutorial(http://rare-technologies.com/word2vec-tutorial/). I trained the model on my lab's server and when I load the model from code on the server it works fine.

But when I downloaded the model on my Windows laptop loading the model gives me the following error: 

c:\python35\lib\site-packages\gensim\utils.py in unpickle(fname)
    910     with smart_open(fname) as f:
    911         # Because of loading from S3 load can't be used (missing readline in smart_open)
--> 912         return _pickle.loads(f.read())
    913 
    914 

ImportError: No module named 'UserList'

Any help would be appreciated.

Regards,

Keshav

[Radim Řehůřek]

Hello Keshav,

that `UserList` doesn't look like anything coming from gensim. I remember something like that in ancient Pythons (2.2?), but I don't see the connection here. This error is new to me.

It's probably something you stored in the object yourself, your own class/module.

Do you remember using such module in your code?

Best,

Radim

[Manoel Vilela]

I have the same traceback after training a Doc2Vec model and loading him. The evil "UserList" module. I read the source of utils.py and doesn't have any entry of this import. And I don't import nothing about this too. Is really obscure. Maybe pickle problem? Any ideas for fixing this?

[Manoel Vilela]

HMMMMMMMMMMM, appears is a cross version problems. I trained the model using python3.4.3 on a ubuntu server and loading the model from server it works fine here too. But on my machine, using the python3.5.1 I have that same issue. 

Em domingo, 6 de março de 2016 20:22:17 UTC-3, Keshav Mathur escreveu:

[Manoel Vilela]

I found here other people with this problem using pickle with distinct python3 versions: https://github.com/mps-youtube/mps-youtube/issues/366

Em domingo, 6 de março de 2016 20:22:17 UTC-3, Keshav Mathur escreveu:

[Radim Řehůřek]

Thanks for investigating Manoel. Some cross-version pickle incompatibility could very well be the reason, yes.

If it's something specific to python3.x, that would also explain why I've never seen it (we rarely use py3k).

Cheers,

Radim

[Keshav Mathur]

Yes that's exactly the same issue as me. I have Python 3.5.0 on my machine and my lab server(where it was trained) has 3.4.3.

Regards,

Keshav

--
You received this message because you are subscribed to a topic in the Google Groups "gensim" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gensim/wA2l4EjlqtA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to gensim+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Manoel Vilela]

So... that is a Python related issue, not about gensim. Other people have the same issue of us: https://bugs.python.org/issue26013.

Somebody talk a workaround using this:

```

But it is too late to fix Python 3.4, it can get only security fixes. You have Celery specific workaround. More general workaround is to update _compact_pickle.IMPORT_MAPPING, e.g.:

import _compact_pickle
_compact_pickle.IMPORT_MAPPING.update({
    'UserDict': 'collections',
    'UserList': 'collections',
    'UserString': 'collections',
    'whichdb': 'dbm',
    'StringIO':  'io',
    'cStringIO': 'io',
})

```

I will try that later, try this too Keshav.

Regards,
Manoel.

-- 
As lembranças não são só do passado. Podem ser de agora e até de amanhã.
— Serial Experiments Lain

[Manoel Vilela]

WOW! Works fine here. My model is loaded with python 3.5.1! : D I'm very happy now.

Ps.: but the name of lib is wrong, change that when calling: _compact_pickle -> _compat_pickle.

On 04/11/2016 02:10 PM, Keshav Mathur wrote:

[Radim Řehůřek]

Great, thanks for reporting Manoel!

That's a nasty, obscure gotcha indeed. I wonder what else broke between the two Python releases, in the same way...

-rr

To unsubscribe from this group and all its topics, send an email to gensim+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "gensim" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gensim/wA2l4EjlqtA/unsubscribe.

To unsubscribe from this group and all its topics, send an email to gensim+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.