Use of Pandas in GAE

[Juan Antonio Fernández Sánchez]

Hi everyone!

I'm trying to run Pandas library in GAE app. I know that it use C internally and I'm wondering if exist any simple solution to execute this is GAE of if I've to think to deploy this microservice in a Compute Engine container.

Any help will be welcome, thank you so much!

[Nick (Cloud Platform Support)]

Hey Juan Antonio, 

As you noted correctly, in the Standard Environment, only pure python apps are supported. However in the Flexible Environment, there is no such restriction, and you can run any python app, regardless of it's use of C libraries! All you'll need to do is go into your python app's app.yaml file and add the line "env: flex". Now, if you've installed the pandas library to a lib/ folder in your app's directory and added the folder to the python import path (sys.path), you'll be able to import pandas without an issue.

And of course, you can use Compute Engine as well, depending on what exactly you'd prefer. What purpose are you deploying machines running pandas for? Maybe I could help if you've got questions about the trade-offs between a Flexible Environment and Compute Engine deployment.

Cheers,

Nick
Cloud Platform Community Support

[Juan Antonio Fernández Sánchez]

Thank you so much for the answer Nick, I'm trying to do this without success.

I've the dependencie in my requirements.txt file and in the module where I import the module like this: import pandas as pd-

If I try to execute the code I've this exception:

ImportError: Missing required dependencies ['numpy']

And if I add numpy in requirements.txt (executing it) and add the module in the same file where I've pandas I've this exception:

ImportError:
Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control).  Otherwise reinstall numpy.

I' m searching solutions in Stack Overflow but I don't find anything interesting. I don't know if you have some simple example where I could see it working.
I would like use only pandas but i don't know if in the future I will need numpy.

Could you help me?

Thank you so mucho, any help will be welcome.

[Nick (Cloud Platform Support)]

Hey Juan,

It could be because in installing the C -based python modules locally, pip is compiling C code for your system's architecture, which wouldn't translate to the deployed machine. If this is the case, see if the following works as a solution:

You can write a Dockerfile for your Custom Runtime app that will install the requirements when the container image is built, that way it'll use the architecture of the container that will actually run the app. Here's an example Dockerfile from our python custom runtime github repository:

FROM gcr.io/google-appengine/python

# Create a virtualenv for dependencies. This isolates these packages from
# system-level packages.
RUN virtualenv /env

# Setting these environment variables are the same as running
# source /env/bin/activate.
ENV VIRTUAL_ENV /env
ENV PATH /env/bin:$PATH

# Copy the application's requirements.txt and run pip to install all
# dependencies into the virtualenv.
ADD requirements.txt /app/requirements.txt
RUN pip install -r /app/requirements.txt

# Add the application source code.
ADD . /app

# Run a WSGI server to serve the application. gunicorn must be declared as
# a dependency in requirements.txt.
CMD gunicorn -b :$PORT main:app

You can see the lines which install dependencies in bold.

Let me know how that works out for you! We could see about getting this caveat for custom runtime deployment and dependencies with c-based python modules documented if that turns out to be the case.

In order for me to be able to test and reproduce what you've seen in future, could you also attach the requirements.txt to your reply?

Cheers,

Nick
Cloud Platform Community Support

[Juan Antonio Fernández Sánchez]

Hi Nick,

Thank you for your answer. The problem is that I would like to run these libraries in a service inside a normal GAE app, not in a container in
Compute Engine.

Thanks!

[Nick (Cloud Platform Support)]

Hey Juan,

In the App Engine Standard Environment, the docs explain that code and libraries must be pure python:

All code for the Python runtime environment must be pure Python, and not include any C extensions or other code that must be compiled. 

However, for the Flexible Environment, you can use C modules, certainly, along with any code that would run on a linux machine. The code runs inside a Docker container, but it's not quite the same as deploying on a Compute Engine instance and is more like programming for App Engine. 

You should check out the quickstart for a Flexible Environment python app and then see about reading my last comment which showed a Dockerfile which would allow you to install libraries on the python instance container image.

Cheers,

Nick
Cloud Platform Community Support

[wesley chun]

This topic came up recently again in Reddit. I answered there as well as this similar Q&A on SO. The bottom line is that this shouldn't be an issue after 2018 when the 2nd-gen App Engine service launched. You don't have to use GAE flexible, and if you think you do, try Cloud Run 1st.

Cheers,

--Wesley

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
"A computer never does what you want... only what you tell it."
    wesley chun :: @wescpy :: Software Architect & Engineer
    Developer Advocate at Google by day; at night: Core Python