Managed VMs - Scipy

Ludovic Legrand

unread,

Feb 23, 2016, 1:48:21 PM2/23/16

to Google App Engine

Hi,

Given discussions in the forums the bug database, I was under the impression I could build a "Managed VM" with packages such scipy.

I am using the python-compat runtime but the deployment does not work, it complains about scipy.

Which runtime should I use?

Do I need to build the vm myself with a Dockerfile?

Thanks for your help.

-------------------------------------------- DOCKER BUILD OUTPUT ---------------------------------------------

Step 0 : FROM gcr.io/google_appengine/python-compat-multicore

---> 2d3ca297268a

Step 1 : ADD . /app/

---> 55bdeb10465e

Removing intermediate container 2f09d79368e4

Step 2 : RUN if [ -s requirements.txt ]; then pip install -r requirements.txt; fi

---> Running in 3aa20701d4f5

Collecting Flask==0.10.1 (from -r requirements.txt (line 1))

Downloading Flask-0.10.1.tar.gz (544kB)

Collecting Scipy==0.17.0 (from -r requirements.txt (line 2))

Downloading scipy-0.17.0.tar.gz (12.4MB)

Requirement already satisfied (use --upgrade to upgrade): Werkzeug>=0.7 in /usr/local/lib/python2.7/dist-packages (from Flask==0.10.1->-r requirements.txt (line 1))

Collecting Jinja2>=2.4 (from Flask==0.10.1->-r requirements.txt (line 1))

Downloading Jinja2-2.8-py2.py3-none-any.whl (263kB)

Collecting itsdangerous>=0.21 (from Flask==0.10.1->-r requirements.txt (line 1))

Downloading itsdangerous-0.24.tar.gz (46kB)

Collecting numpy>=1.6.2 (from Scipy==0.17.0->-r requirements.txt (line 2))

Downloading numpy-1.10.4.tar.gz (4.1MB)

Collecting MarkupSafe (from Jinja2>=2.4->Flask==0.10.1->-r requirements.txt (line 1))

Downloading MarkupSafe-0.23.tar.gz

Installing collected packages: MarkupSafe, Jinja2, itsdangerous, Flask, numpy, Scipy

Running setup.py install for MarkupSafe: started

Running setup.py install for MarkupSafe: finished with status 'done'

Running setup.py install for itsdangerous: started

Running setup.py install for itsdangerous: finished with status 'done'

Running setup.py install for Flask: started

Running setup.py install for Flask: finished with status 'done'

Running setup.py install for numpy: started

Running setup.py install for numpy: still running...

Running setup.py install for numpy: finished with status 'done'

Running setup.py install for Scipy: started

Running setup.py install for Scipy: finished with status 'error'

Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-s7Bw41/Scipy/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-cpy6Ql-record/install-record.txt --single-version-externally-managed --compile:

lapack_opt_info:

openblas_lapack_info:

libraries openblas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']

NOT AVAILABLE

lapack_mkl_info:

mkl_info:

libraries mkl,vml,guide not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']

NOT AVAILABLE

atlas_3_10_threads_info:

Setting PTATLAS=ATLAS

libraries tatlas,tatlas not found in /usr/local/lib

libraries lapack_atlas not found in /usr/local/lib

libraries tatlas,tatlas not found in /usr/lib

libraries lapack_atlas not found in /usr/lib

libraries tatlas,tatlas not found in /usr/lib/x86_64-linux-gnu

libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu

NOT AVAILABLE

atlas_3_10_info:

libraries satlas,satlas not found in /usr/local/lib

libraries lapack_atlas not found in /usr/local/lib

libraries satlas,satlas not found in /usr/lib

libraries lapack_atlas not found in /usr/lib

libraries satlas,satlas not found in /usr/lib/x86_64-linux-gnu

libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu

NOT AVAILABLE

atlas_threads_info:

Setting PTATLAS=ATLAS

libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib

libraries lapack_atlas not found in /usr/local/lib

libraries ptf77blas,ptcblas,atlas not found in /usr/lib

libraries lapack_atlas not found in /usr/lib

libraries ptf77blas,ptcblas,atlas not found in /usr/lib/x86_64-linux-gnu

libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu

NOT AVAILABLE

atlas_info:

libraries f77blas,cblas,atlas not found in /usr/local/lib

libraries lapack_atlas not found in /usr/local/lib

libraries f77blas,cblas,atlas not found in /usr/lib

libraries lapack_atlas not found in /usr/lib

libraries f77blas,cblas,atlas not found in /usr/lib/x86_64-linux-gnu

libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu

NOT AVAILABLE

/usr/local/lib/python2.7/dist-packages/numpy/distutils/system_info.py:1552: UserWarning:

Atlas (http://math-atlas.sourceforge.net/) libraries not found.

Directories to search for the libraries can be specified in the

numpy/distutils/site.cfg file (section [atlas]) or by setting

the ATLAS environment variable.

warnings.warn(AtlasNotFoundError.__doc__)

lapack_info:

libraries lapack not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']

NOT AVAILABLE

/usr/local/lib/python2.7/dist-packages/numpy/distutils/system_info.py:1563: UserWarning:

Lapack (http://www.netlib.org/lapack/) libraries not found.

Directories to search for the libraries can be specified in the

numpy/distutils/site.cfg file (section [lapack]) or by setting

the LAPACK environment variable.

warnings.warn(LapackNotFoundError.__doc__)

lapack_src_info:

NOT AVAILABLE

/usr/local/lib/python2.7/dist-packages/numpy/distutils/system_info.py:1566: UserWarning:

Lapack (http://www.netlib.org/lapack/) sources not found.

Directories to search for the sources can be specified in the

numpy/distutils/site.cfg file (section [lapack_src]) or by setting

the LAPACK_SRC environment variable.

warnings.warn(LapackSrcNotFoundError.__doc__)

NOT AVAILABLE

Running from scipy source directory.

Traceback (most recent call last):

File "<string>", line 1, in <module>

File "/tmp/pip-build-s7Bw41/Scipy/setup.py", line 265, in <module>

setup_package()

File "/tmp/pip-build-s7Bw41/Scipy/setup.py", line 262, in setup_package

setup(**metadata)

File "/usr/local/lib/python2.7/dist-packages/numpy/distutils/

core.py", line 135, in setup

config = configuration()

File "/tmp/pip-build-s7Bw41/Scipy/setup.py", line 182, in configuration

config.add_subpackage('scipy')

File "/usr/local/lib/python2.7/dist-packages/numpy/distutils/misc_util.py", line 1002, in add_subpackage

caller_level = 2)

File "/usr/local/lib/python2.7/dist-packages/numpy/distutils/misc_util.py", line 971, in get_subpackage

caller_level = caller_level + 1)

File "/usr/local/lib/python2.7/dist-packages/numpy/distutils/misc_util.py", line 908, in _get_configuration_from_setup_py

config = setup_module.configuration(*args)

File "scipy/setup.py", line 15, in configuration

config.add_subpackage('linalg')

File "/usr/local/lib/python2.7/dist-packages/numpy/distutils/misc_util.py", line 1002, in add_subpackage

caller_level = 2)

File "/usr/local/lib/python2.7/dist-packages/numpy/distutils/misc_util.py", line 971, in get_subpackage

caller_level = caller_level + 1)

File "/usr/local/lib/python2.7/dist-packages/numpy/distutils/misc_util.py", line 908, in _get_configuration_from_setup_py

config = setup_module.configuration(*args)

File "scipy/linalg/setup.py", line 20, in configuration

raise NotFoundError('no lapack/blas resources found')

numpy.distutils.system_info.NotFoundError: no lapack/blas resources found

----------------------------------------

Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-s7Bw41/Scipy/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-cpy6Ql-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-s7Bw41/Scipy

The command [/bin/sh -c if [ -s requirements.txt ]; then pip install -r requirements.txt; fi] returned a non-zero code: 1

--------------------------------------------------------------------------------------------------------------

Beginning teardown of remote build environment (this may take a few seconds).

ERROR: (gcloud.preview.app.deploy) Docker build aborted: The command [/bin/sh -c if [ -s requirements.txt ]; then pip install -r requirements.txt; fi] returned a non-zero code: 1

Deleted [https://www.googleapis.com/compute/v1/projects/quant-data-vision/zones/us-central1-f/instances/gae-builder-vm-20160223t101428].

Jon Parrott

unread,

Feb 23, 2016, 7:26:13 PM2/23/16

to Google App Engine

Hi Ludovic,

Using SciPy on MVMs is absolutely possible. You have a few options:

You can try using the new Managed VMs Python Runtime, which should allow you to just add scipy to your requirements.txt (please let me know if it doesn't!). Keep in mind that the runtime doesn't currently provide access to "google.appengine.*" APIs, as noted here.
If you want to continue to use the compat runtime, you can generate a custom runtime from it and modify the Dockerfile to install any additional system-level resources that you need.

Let me know which one you choose and if you run into any issues.

Ludovic Legrand

unread,

Feb 24, 2016, 1:55:24 AM2/24/16

to Google App Engine

Hi Jon,

Thanks for the prompt reply.

I tried option 1 with:

App.yaml

runtime: python-compat

vm: true

threadsafe: true

handlers:

- url: .* # This regex directs all routes to main.app

script: main.app

requirements.txt

Flask==0.10.1

Scipy==0.17.0

And the message pasted in the original message is the error message I am getting.

Ludo

Jon Parrott

unread,

Feb 24, 2016, 1:44:42 PM2/24/16

to Google App Engine

I'll need to file a bug to add scipy dependencies to the runtime.

In the meantime, you can use this dockerfile: https://gist.github.com/jonparrott/1a58e2f0b4941647f969

You'll need to set runtime to custom in app.yaml, and you don't need to specify scipy in your requirements.txt.

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/Uy-qHkE5-Ig/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/0d416b65-ed4d-4591-8bfd-ba8bb6dca3c2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply all

Reply to author

Forward