Running gensim on aws lambda

[Sam Sam]

Hi,

 I am trying to run Gensim Version 2.2.0 on AWS lambda and got the following error. I have installed cython and have numpy/scipy libraries compatabile to aws instance. Any suggestions/steps would help. 

[u'hi']
[WARNING]	2017-07-16T19:06:57.158Z-6a59-11e7-8325-cf5443db16b6	Slow version of gensim.models.word2vec is being used
/var/task/gensim/models/word2vec.py:787: UserWarning: C extension not loaded for Word2Vec, training will be slow. Install a C compiler and reinstall gensim for fast training.
  warnings.warn("C extension not loaded for Word2Vec, training will be slow. "
END RequestId: 6a59-11e7-8325-cf5443db16b6
REPORT RequestId: ecfb3911-6a59-11e7-8325-cf5443db16b6	Duration: 4003.68 ms	Billed Duration: 4000 ms 	Memory Size: 128 MB	Max Memory Used: 49 MB	 

[Sam Sam]

Forgot to mention basically the function times out without running to completion and I am using Python 2.7

[Gordon Mohr]

Given AWS Lambda's strict timeouts, have you verified the same configuration & task will run to completion in a more typical environment first?

The error about the C-extension not being loaded doesn't require `cython` to be installed, but rather for full python/OS build tools to be available at the time gensim is installed. On Ubuntu, this usually means at least the `build-essentials' and 'python-dev' packages are installed before installing gensim. Essentially, watch the logs/errors very carefully when installing gensim & numpy/scipy, and address any problems indicated there before trying to use gensim.

- Gordon

[Sam Sam]

Thank you. On my local MacOS it works perfectly fine. The numpy/scipy libraries are also correct since I use it for another lambda function and I complied it on a ec2 instance. 

Do I need to install gensim on ec2 instance and copy the libraries from there?  

[Gordon Mohr]

Unless AWS Lambda is running OSX, a test on OSX won't be a good way to verify/debug proper operations on AWS Lambda. 

You'll have to check the AWS Lambda docs/support-forums to learn how software is installed on those systems,or the installations monitored for correctable errors. 

- Gordon

[Sam Sam]

Hi,

 Thank you. Fixed it. Basically  we have to install gensim on a ec2 AMI linux instance and copy libraries into lambda. When we do that it works perfectly.