cannot reshape array of size 493384700 into shape (2000005,300)

[emmanuel chappat]

Hi, 

I am trying to load a fastText model saved using Gensim `save` on a Sagemake inference instance using python 3.7. The instantiation fails with the followin log in cloudwatch:

ValueError: cannot reshape array of size 493384700 into shape (2000005,300)

Then later in the same logs I am getting this time a different shape.

ValueError: cannot reshape array of size 18113532 into shape (2000010,300)

Does that ring any bell ?

Thanks a lot,

Emmanuel

[Gordon Mohr]

What is the code, including any file paths used, that triggers the error? 

Do you see and can you share the full error stack, showing each line of code involved, rather than just the headline error?

A float array of size (2000005,300) would require 600,001,500 floats (about 2.4 GB). So, the 1st error you're showing, mentioning an array of size 493,384,700 elements, is missing about 106,000,000 necessary values. Are you sure the prior `save()` succeeded in full, and the related files were never truncated? Can you try the training/save again, and watch it closely to be sure it succeeds and creates files of appropriate sizes to reflect the full model.

Notably, (493/600) * 2.4GB = 1.97GB, suspiciously close to the 2GB maximum value of signed 32-bit ints, and implementation limits in certain 32-bit executables/OSes/filesystems. So at least for the 1st error, it's possible such implementation limits have been hit somewhere in your process.

- Gordon

[Radim Řehůřek]

There was a similar issue recently here: https://github.com/RaRe-Technologies/gensim/issues/2473 (a bug in NumPy)

In case that's the problem you're hitting too, check out the resolution in the comments.

HTH,

Radim

[emmanuel chappat]

Thanks Gordon, that makes sense. I am gonna try training the model again.

[emmanuel chappat]

Thanks Radim, I did stumble on that thread but it does not seem to be apply in my instance as the model is in Gensim format and already decompressed.