py4web on google app engine

[Soporte MisDatos]

Can I deploy all py4web on google app engine?

I try to follow py4web\deployment_tools\gae\Makegilr's instructions (from windows) and end up with a folder like this. 

gae

   apps (copy and paste from all py4web\apss

   lib (copy an paste from py4web\py4web

   index 

   app.yaml (copy and paste from  py4web\deployment_tools\gae)

   requeriments (idem)

    main (idem)

but when I deploy and open the url in the console I receive the nginx 502 bad gateway error and the reporting error traceback to arbiter.py

thanks for any help

[Luca de Alfaro]

Yes, I use py4web on appengine for most of my "serious" work. 

You need to look in the logs, they will offer some clue about what is causing the error.  Some things I have run into: 

• The .service file is missing
• You cannot write to the filesystem; in particular, you need to use a real database (not sqlite), and you need to configure it yourself and use migrate=False. 

I put up some of these considerations in here: Unit 20.  It will take you a bit of time, but once it's up and running, it works extremely well.  My newest creation with it is notebookgrader.com :-) 

Luca

--
You received this message because you are subscribed to the Google Groups "py4web" group.
To unsubscribe from this group and stop receiving emails from it, send an email to py4web+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/py4web/439a370d-83a0-43c8-8173-a95a44561ca2n%40googlegroups.com.

[Soporte MisDatos]

thanks a lot!. I will take it into account. anyway, I leave more information about the error here in case it helps.

from py4web I just try to run the example app _minimal\index. (I understand that it does not use data, but I will check)}

Actually I have a small application in we2py (I love web2py because it allowed me to implement a web it almost without knowing python) but it is deployed on gae in python2 and is being deprecate by goolge. And I am trying to update to web2py with python3 or, failing that, py4web. in both I get the reference error. 

Going back to py4web's point, the error log in google app engine is (trying run py4web as it comes in its source code, without changes, only those listed in ...developer_tools\gae\makefile...) when browsing

https:/****appspot.com/

I see: 502 Bad Gateway

nginx

and gogole app engine reporting error:

self.wsgi = self.app.wsgi():
at .load_wsgi ( /layers/google.python.pip/pip/lib/python3.11/site-packages/gunicorn/workers/base.py:146 )
at .init_process ( /layers/google.python.pip/pip/lib/python3.11/site-packages/gunicorn/workers/base.py:134 )
at .spawn_worker ( /layers/google.python.pip/pip/lib/python3.11/site-packages/gunicorn/arbiter.py:609 )

I don't know much about python but I suspect it's a problem in the app.yaml file

(https://cloud.google.com/appengine/docs/standard/python3/runtime?hl=es-419#application_startup, sorry in spanish I hope I can find the equivalent in English)

thank you in advance

[Jacinto Parga]

Hi, 

I'm deploying apps in google app engine and they work fine.

This is an example of app.yaml that works for me:

service: yourservice
runtime: python311

# Handlers define how to route requests to your application.
handlers:
  - url: .* # This regex directs all routes to main.bottle
    script: auto
    secure: always
    redirect_http_response_code: 301

  - url: /static
    static_dir: static

  - url: /favicon.ico
    static_files: static/favicon.ico
    upload: static/favicon.ico

  - url: /robots.txt
    static_files: static/robots.txt
    upload: static/robots.txt

I usually have  issues with the requirements.txt file that makes the app very big. Sometimes I include the libraries in the /lib folder.

Another source of nasty job is to create manually the db. Web2py did it great and even using google datastore.

Google datastore is not plenty available with pydal. There is a branch in git repository than allows you to deploy a app using datastore as db, but it is not completely reliable yet. It is a pity because for applications with a low use it is a cheaper solution than a SQL db.

[Soporte MisDatos]

thank you! but the error continues. 

I try this:

download source code py4web from https://github.com/web2py/py4web.git (download to my disk c)

download source code python from https://www.python.org/downloads/ (idem, download to my disk c)

then I run locally py4web:

   python3 py4web.py setup apps

python3 py4web.py set_passwordpython3 py4web.py run appsI browse locally py4web with no error.then I want run py4web on google app engine just as it isCopy the example files from py4web /path/to/py4web/development_tools/gcloud/* (Actually /path/to/py4web/development_tools/gae) to new folder c:\gae
copy C:\py4web\py4web to gae\libs\py4webcopy py4web\apps to gae\apps.update requeriments.txt (attach)rest of file (app.yaml, main.py....) remain unchangedthen donwload gcloud SDK.login in my account and from c:\gae I rungcloud app deploy app.yaml --version NNupload with no errorthen I browse the url in the console (https:\\**appspot) I receive the nginx 502 bad gateway error and the reporting error traceback to arbiter.py
all help is welcome

[Luca de Alfaro]

In my experience, it is absolutely impossible to debug these issues without looking at the log output. 

There can be a zillion little things that do not work, and one has to look at the logs and solve them one by one; you cannot do this blind. 

So please head to the logging console, and tell us what you see there. 

Luca

To view this discussion on the web visit https://groups.google.com/d/msgid/py4web/a49e519d-d4ba-4874-994a-fa9efbb0ebfbn%40googlegroups.com.

[Soporte MisDatos]

Yes yes of course, in the browse

https://nn-dot-xxxxx.uc.r.appspot.com/

502 Bad Gateway
nginx

in reporting error gae console

Traceback (most recent call last):
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
    worker.init_process()
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/gunicorn/workers/gthread.py", line 95, in init_process
    super().init_process()
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/gunicorn/workers/base.py", line 134, in init_process
    self.load_wsgi()
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
    self.wsgi = self.app.wsgi()

the main.py is    py4web\deployment_tools\gae unchanged:

import os
import site
site.addsitedir(os.path.join(os.path.dirname(__file__), 'lib'))
from py4web.core import Reloader, bottle, Session
os.environ['PY4WEB_DASHBOARD_MODE'] = 'none'
os.environ['PY4WEB_SERVICE_DB_URI'] = 'sqlite:memory'
os.environ['PY4WEB_APPS_FOLDER'] = os.path.join(os.path.dirname(__file__), 'apps')
os.environ['PY4WEB_SERVICE_FOLDER'] = os.path.join(os.path.dirname(__file__), '.service')
Session.SECRET = open(os.path.join(os.path.dirname(__file__), '.service/session.secret'), 'rb').read()
Reloader.import_apps()
app = bottle.default_app()

I hope this helps

[Luca de Alfaro]

Hmm, nothing of that points to the error... do you have other output in the logs nearby, before or after? 

Luca

To view this discussion on the web visit https://groups.google.com/d/msgid/py4web/CAF2BMT85Veep9nWi1Hk5SxH-UQAT3R6_eYRsJiHHWSxU3AJtcA%40mail.gmail.com.

[Jacinto Parga]

Try with this requirements.txt, with fevent and gunicorn disabled.

#
# Ojo poner sólo lo que se use
#
wheel >= 0.37.1
ombott >= 1.0.0
click >= 8.1.3
colorama >= 0.4.5
cryptography >= 37.40.4
portalocker >= 2.5.1
tornado >= 6.2
renoir >= 1.5.4
requests >= 2.31.0
threadsafevariable >= 1.2
pyjwt >= 2.0.1
pycryptodome >= 3.15.0
pluralize >= 0.1.9
rocket3 >= 20230507.1
yatl >= 20230507.3
pydal >= 20230521.1
watchgod >= 0.6
urllib3  >=1.26.11

# optional modules:
# gunicorn
# gevent
# gevent-websocket
# python-memcached # used for testing or for optionally actually accessing memcache

#Para la base de datos
mysqlclient

[Soporte MisDatos]

Thanks. I am going to try.  but now I see maybe my problem

If I try to deploy (on google app engine)  the py4web as it is downloaded with its applications (_minimal, etc.). py4web will use a database? py4web running in gae  automatically will use google:datastore?. because the main.py in py4web\developer_tools\gae\main.py is:

import os
import site
site.addsitedir(os.path.join(os.path.dirname(__file__), 'lib'))
from py4web.core import Reloader, bottle, Session
os.environ['PY4WEB_DASHBOARD_MODE'] = 'none'
os.environ['PY4WEB_SERVICE_DB_URI'] = 'sqlite:memory'
os.environ['PY4WEB_APPS_FOLDER'] = os.path.join(os.path.dirname(__file__), 'apps')
os.environ['PY4WEB_SERVICE_FOLDER'] = os.path.join(os.path.dirname(__file__), '.service')
Session.SECRET = open(os.path.join(os.path.dirname(__file__), '.service/session.secret'), 'rb').read()
Reloader.import_apps()
app = bottle.default_app()

or I need a connection to  mysql or somelse?

in web2py on gae (python2) I deployed it directly unchanged because web2py automatically used google:datastore as database (for data session, etc.)

remember in google app engine no file system is allowed.

Thanks for all.

You received this message because you are subscribed to a topic in the Google Groups "py4web" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/py4web/SdSVsFyXGDI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to py4web+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/py4web/61e989e2-c12a-4538-bf51-4793e4809a06n%40googlegroups.com.

[Luca de Alfaro]

I always use a Google Cloud SQL instance; the cheapest one is about $8/month.  And it allows all kind of queries, whereas I find datastore to be incredibly limited (no joins, no inequalities on more than 1 field, ...). 

Luca

To view this discussion on the web visit https://groups.google.com/d/msgid/py4web/CAF2BMT-Ct6Kr_-see4Wr2o0dMF%2B%2BY1qwVzmuAYAaT0Lee49hzg%40mail.gmail.com.

[Luca de Alfaro]

Also SQL is standard; if you use google datastore, good luck getting your data out if you want to change hosting platform. 

Luca

[Soporte MisDatos]

Sure. I understand.

My intention for now is just to deploy "hello word" in py4web on google app engine. using standar apps included in the py4web.

based on what jacinto said My Doubt is: py4web running on gae use google:datastore automatically  for your internal data (session, ticket, etc.)?

[Luca de Alfaro]

No, you would have to define it yourself.  I have this in my common.py : https://github.com/lucadealfaro/notebookgrader-app/blob/main/common.py#L44 

You would have to define your own db = DAL(...) line. There is nothing automatic. 

Luca

To view this discussion on the web visit https://groups.google.com/d/msgid/py4web/CAF2BMT-5WCyTEy47OwB_1iDGCqtkx2AdfuKSjmPA27TS2ZqDOQ%40mail.gmail.com.

[Jacinto Parga]

There is more than one thing that you have to consider.

You cannot write in the file system so you cannot use required_folder so you have to change the settings of the app like this

This file is provided as an example:
"""
import os

# from py4web.core import required_folder

# db settings
APP_FOLDER = os.path.dirname(__file__)
APP_NAME = os.path.split(APP_FOLDER)[-1]
# DB_FOLDER:    Sets the place where migration files will be created
#               and is the store location for SQLite databases
DB_FOLDER = os.path.join(APP_FOLDER, "databases")

If you want to use the Datastore (I like it for some applications) there is a non stable and reliable version of pydal that allows it: https://github.com/web2py/pydal/tree/new-google-datastore

I have not the time/resources/talent to improve it. You know there was a discussion about it: https://groups.google.com/g/py4web/c/v9P44ZL4lOI/m/b6QA4zmiAQAJ

But in a way it works, I have deployed a simple app with it.  https://mereba.com  You can even login if you have a gmail.com address. Forms work fine, Grids don't. I use Datatables instead of Grid.

You cannot make all the queries like in web2py due to the joins and the or queries. In web2py the auth libraries work perfect with Datastore, but in py4web auth.py and some other utils do not prevent the use of that type of queries.

Here is the settings configuration to access the DB: 

DB_FOLDER = os.path.join(APP_FOLDER, "databases")

# Separamos entorno de producción del local de desarrollo

if os.environ.get("GAE_ENV"):
    DB_URI = "google:datastore"
    DB_POOL_SIZE = 1
    DB_MIGRATE = True
    DB_FAKE_MIGRATE = False  # maybe?
else:
    DB_URI = "sqlite://storage.db"
    DB_POOL_SIZE = 1
    DB_MIGRATE = True
    DB_FAKE_MIGRATE = False  # maybe?

# location where static files are stored:
STATIC_FOLDER = os.path.join(APP_FOLDER, "static")

# location where to store uploaded files:
UPLOAD_FOLDER = os.path.join(APP_FOLDER, "uploads")

Luca is right Datastore is not the best option to have a more standar access to the data. Whenever I want to expor them I have to use hand made API REST.  Py4web is good at API.

But I liked Datastore for simple apps because (I think) it is very robust in terms of security, I do not know of common threats that directly attack data in Datastore as there are for other databases.

Unfortunatly I can not use Datastore for "serious apps" with py4web yet. 

[Soporte MisDatos]

good.

but then, the file main.py cannot be used as is in git?

https://github.com/web2py/py4web/blob/master/deployment_tools/gae/main.py

The code you see there is used to execute in GAE?

or I must to replace nos.environ['PY4WEB_SERVICE_DB_URI'] = 'sqlite:memory' by os.environ['PY4WEB_SERVICE_DB_URI'] = 'google:datastore'

this steps listed in https://github.com/web2py/py4web/blob/master/deployment_tools/gae/README.md

These are all the necessary steps to run the py4web (only with default apps, without bases or anything else, just "hello word") on GAE?

thanks for all

[Jacinto Parga]

Yes the git instructions work. I have made some adaptations, but the https://github.com/web2py/py4web/blob/master/deployment_tools/gae/  are correct.

They work fine with de last pydal and SQL databases (Mysql or Postrgresql), I have tested them.

Datastore requires the https://github.com/web2py/pydal/tree/new-google-datastore that I include in the /lib directory like py4web and remove it from the requirements.

This is my main.py

import os

import site

import os

import site

""" This is the main.py file according to deployments/tools with some addings to allow local testing.

Can be used in local debugging replacing py4web.py

"""

site.addsitedir(os.path.join(os.path.dirname(__file__), "lib"))

# This is for finding the path of py4web in GAE

from py4web.core import Reloader, bottle, Session

os.environ["PY4WEB_DASHBOARD_MODE"] = "full" # or 'demo' or 'full'

os.environ["PY4WEB_SERVICE_DB_URI"] = "google:datastore" # "sqlite:memory" #

os.environ["PY4WEB_APPS_FOLDER"] = os.path.join(os.path.dirname(__file__), "apps")

os.environ["PY4WEB_SERVICE_FOLDER"] = os.path.join(

os.path.dirname(__file__), "apps/.service"

)

Session.SECRET = open(

os.path.join(os.path.dirname(__file__), "apps/.service/session.secret"), "rb"

).read()

# For deploying in GAE

if os.getenv("GAE_ENV", "").startswith("standard"):

# Production in the standard environment

Reloader.import_apps()

app = bottle.default_app()

# For local testing

else:

# Local execution.

if __name__ == "__main__":

# This is used when running locally only. When deploying to Google App

# Engine, a webserver process such as Gunicorn will serve the app. This

# can be configured by adding an `entrypoint` to app.yaml.

# App Engine itself will serve those files as configured in app.yaml.

# app.run(

# host="127.0.0.1", port=9000, debug=True

# ) # Also works but does not allow _dashboard

os.system("py4web run apps -P 9000")

And the requirements.txt (it is dirty because it is an old file for testing.)

wheel

ombott >= 0.0.10

click

colorama

cryptography

portalocker

tornado

renoir >= 1.4.0

requests

threadsafevariable >= 1.2

pyjwt >= 2.0.1

pycryptodome

pluralize >= 0.1.9

rocket3

yatl >= 20211217.1

#pydal >= 20210626.3 # Incluimos la versión para datastore en lib

watchgod >= 0.6

# optional modules:

# gunicorn

# gevent

# gevent-websocket

# python-memcached # used for testing or for optionally actually accessing memcache

# Esto es para depurar el local

# Las podemos poner en lib

google-cloud-datastore

# google-cloud-storage

# google-python-cloud-debugger

# google-cloud-recaptcha-enterprise # Es necesario un lio con las dependencias

# google.cloud

# Sólo para la implementación en local

# gunicorn

[Jacinto Parga]

this is an example with datastore https://best-try.appspot.com/best 

[Soporte MisDatos]

Resolved!

Thank you all especially Jacinto

I have written down the steps to deploy  py4web (as downloaded from its source code with no database) from Windows to google app engine

download python311 https://www.python.org/downloads/
download py4web source code https://github.com/web2py/py4web
go to folder py4web
c:\python311\python -m pip install  --upgrade -r requirements.txt
python3 -m pip install --upgrade py4web
to run locally  py4web according to https://py4web.com/_documentation/static/en/chapter-03.html#first-run
to copy folder C:\py4web\deployment_tools\gae to c:\gae
to create a subfolder c:\gae\lib
to copy on it py4web\py4web to c:\gae\lib\py4web
to copy on it py4web\py4web\apss to c:\gae\lib\apps
to extract pydal de https://github.com/web2py/pydal/tree/new-google-datastore  and  copy to c:\gae\lib\pydal
attention!  to replace  c:\gae\main.py  with suggested by jacinto (attach)
attention!  to replace   c:\gae\requeriments.txt por  with suggested by jacinto (attach)
download  gcloud sdk https://cloud.google.com/sdk/docs/downloads-interactive?hl=es-419  or update gcloud components update

on folder c:\gae
run sdk gcloud app deploy --version NN