forms.py error when deploying from main.py

[Jacinto Parga]

This is the controller: 

@unauthenticated()

@action("cosa", method=["POST", "GET"])

@action.uses(db, session, "generic.html")

def cosa():

form = Form(db.thing)

rows = db(db.thing).select()

return dict(form=form, rows=rows)

It works fine when running the app form _dashboard, but when running from main.py (deployment_tools/gae). this is the error:

ERROR:root:Traceback (most recent call last):

  File "/home/jacinto/.pyenv/versions/py4web382/lib/python3.8/site-packages/py4web/core.py", line 722, in wrapper

    ret = func(*func_args, **func_kwargs)

  File "/home/jacinto/.pyenv/versions/py4web382/lib/python3.8/site-packages/py4web/core.py", line 682, in wrapper

    ret = func(*args, **kwargs)

  File "/home/jacinto/PROGRAMACION/TESTS/py4webtest/deploygae/apps/_default/controllers.py", line 161, in cosa

    form = Form(db.thing)

  File "/home/jacinto/.pyenv/versions/py4web382/lib/python3.8/site-packages/py4web/utils/form.py", line 479, in __init__

    self._sign_form()

  File "/home/jacinto/.pyenv/versions/py4web382/lib/python3.8/site-packages/py4web/utils/form.py", line 512, in _sign_form

    self.formkey = jwt.encode(payload, self._get_key(), algorithm="HS256").decode(

  File "/home/jacinto/.pyenv/versions/py4web382/lib/python3.8/site-packages/py4web/utils/form.py", line 504, in _get_key

    return key + "." + json.dumps(additional_info)

TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

The rest of the application works, even register and login.

[Massimo]

For now use Form(...., csrf_session=session)

The problem is that otherwise Form does not know you have a session. It defaults to use a global Session.SECRET which is not set on GAE because not deployed since the FS is readonly.

Need to think of a better way to make this work out of the box.

[Massimo]

I have a possible solution for this (in master) but there is a bigger problem.

Before you deploy on GAE you must run at least once locally. It will create a file

apps/.service/session.secret

This is  UUID that is used to encrypt sessions. You must make sure it is deployed on GAE along with the other files.

On Saturday, 5 September 2020 at 05:56:24 UTC-7 jpa...@gmail.com wrote:

[Jacinto Parga]

Thanks Massimo.

1 -   Form(...., csrf_session=session)   works perfect in local execution out of the box and in GAE. 

It is different because in local execution py4web is gotten  from the python local interpreter, but in GAE is gotten from the previously installed in .lib/py4web 

Both works

2- I make it run  locally from _dahsboard and the apps/.service/session.secret was created, but if I remove csrf_session=session it fails again in local execution from main.py and in GAE, although apps/.service/session.secret was deployed

Only works from _dashboard when removing csrf_session=session

3 - The issue also happens with Grid create form, as I cannot pass the session argument to the form:

ERROR:root:Traceback (most recent call last):

  File "/home/jacinto/.pyenv/versions/py4web382/lib/python3.8/site-packages/py4web/core.py", line 722, in wrapper

    ret = func(*func_args, **func_kwargs)

  File "/home/jacinto/.pyenv/versions/py4web382/lib/python3.8/site-packages/py4web/core.py", line 684, in wrapper

    ret = obj.transform(ret, shared_data)

  File "/home/jacinto/.pyenv/versions/py4web382/lib/python3.8/site-packages/py4web/core.py", line 423, in transform

    output = yatl.render(

  File "/home/jacinto/.pyenv/versions/py4web382/lib/python3.8/site-packages/yatl/template.py", line 968, in render

    exec(code, context)

  File "<string>", line 35, in <module>

  File "/home/jacinto/.pyenv/versions/py4web382/lib/python3.8/site-packages/yatl/template.py", line 831, in write

    data = data.xml()

  File "/home/jacinto/.pyenv/versions/py4web382/lib/python3.8/site-packages/py4web/utils/grid.py", line 119, in xml

    return self.make().xml()

  File "/home/jacinto/.pyenv/versions/py4web382/lib/python3.8/site-packages/py4web/utils/grid.py", line 131, in make

    form = Form(table, record, deletable=self.deletable, **self.form_attributes)

  File "/home/jacinto/.pyenv/versions/py4web382/lib/python3.8/site-packages/py4web/utils/form.py", line 479, in __init__

    self._sign_form()

  File "/home/jacinto/.pyenv/versions/py4web382/lib/python3.8/site-packages/py4web/utils/form.py", line 512, in _sign_form

    self.formkey = jwt.encode(payload, self._get_key(), algorithm="HS256").decode(

  File "/home/jacinto/.pyenv/versions/py4web382/lib/python3.8/site-packages/py4web/utils/form.py", line 504, in _get_key

    return key + "." + json.dumps(additional_info)

TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

I tried it with apps/.service/session.secret previously created.

[Massimo DiPierro]

With the new change you should not need that any more.

-- 
You received this message because you are subscribed to the Google Groups "py4web" group.
To unsubscribe from this group and stop receiving emails from it, send an email to py4web+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/py4web/b4ffd654-51a7-4a7e-ba5b-5895675df2f7n%40googlegroups.com.

[Jacinto Parga]

Ok Thanks

I dont understand why auth forms worked and Form doesn’t. 

[Massimo]

Auth is from a single page web app so it uses APIs and does not need CRSF protection.

The real issue is, are you session cookies encrypted or not? They should be but they may be not if no Session.SECRET.

That would be a vulnerability that needs to be prevented. Will make it a priority to investigate and resolve.

This would only be a problem on GAE if the secret is not deployed.

[Massimo]

Do you have a skip files in your app.yaml? You may be preventing deployment of apps/.service_folder/session.secret

On Saturday, 5 September 2020 at 05:56:24 UTC-7 jpa...@gmail.com wrote:

[Jacinto Parga]

No, the .apps/service/session.secret is deployed in GAE:

In GAE with python 3 the files are skipped whith a .gcloudignore file, like .gitignore in git repositories, not in app.yaml

The thing is that session.secret file is created when tha app is executed from _dahsboard but not when it is executed from main.py if local testing or GAE. I am using this main.py file useful for local debugging:

import os

import site

site.addsitedir(os.path.join(os.path.dirname(__file__), "lib"))

from py4web.core import Reloader, bottle

os.environ["PY4WEB_DASHBOARD_MODE"] = "demo"

os.environ["PY4WEB_SERVICE_DB_URI"] = "sqlite:memory"

os.environ["PY4WEB_APPS_FOLDER"] = os.path.join(os.path.dirname(__file__), "apps")

os.environ["PY4WEB_SERVICE_FOLDER"] = os.path.join(

    os.path.dirname(__file__), "apps/.service"

)

Reloader.import_apps()

app = bottle.default_app()

#  For local test

if os.getenv("GAE_ENV", "").startswith("standard"):

    # Production in the standard environment

    MENSAJE = "do whatever"

else:

    # Local execution.

    if __name__ == "__main__":

        # This is used when running locally only. When deploying to Google App

        # Engine, a webserver process such as Gunicorn will serve the app. This

        # can be configured by adding an `entrypoint` to app.yaml.

        # App Engine itself will serve those files as configured in app.yaml.

        app.run(host="127.0.0.1", port=9000, debug=True)

And this is the 500 error in GAE when CREATE FORM from grid is used or a Form without csft_session=session:

2020-09-08 14:02:27.145 CEST{"textPayload":"","insertId":"5fXXXXXXXXXXXXXXXXXXXX","resource":{"type":"gae_app","labels":{"module_id":"py4webdatastore","version_id":"2","project_id":"web2gae","zone":"us14"}},"timestamp":"2020-09-08T12:02:27.145260Z","labels":{"clone_id":"0XXXXXXXXXXXXXXXXXXXXXXXXXX

2020-09-08 14:02:27.145 CESTTypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

2020-09-08 14:02:27.145 CESTreturn key + "." + json.dumps(additional_info)

2020-09-08 14:02:27.145 CESTFile "/srv/lib/py4web/utils/form.py", line 504, in _get_key

2020-09-08 14:02:27.145 CESTself.formkey = jwt.encode(payload, self._get_key(), algorithm="HS256").decode(

2020-09-08 14:02:27.145 CESTFile "/srv/lib/py4web/utils/form.py", line 512, in _sign_form

2020-09-08 14:02:27.145 CESTself._sign_form()

2020-09-08 14:02:27.145 CESTFile "/srv/lib/py4web/utils/form.py", line 479, in __init__

2020-09-08 14:02:27.145 CESTform = Form(

2020-09-08 14:02:27.145 CESTFile "/srv/lib/py4web/utils/grid.py", line 132, in make

2020-09-08 14:02:27.145 CESTreturn self.make().xml()

2020-09-08 14:02:27.145 CESTFile "/srv/lib/py4web/utils/grid.py", line 119, in xml

[Jacinto Parga]

May be this help

I have seen that session form common.py and Session from py4web core.py  are treated in a different way if the execution is from _dahsboard or out of the box:

Test controller: 

@action("controlsesion")

@action.uses(session, "control.html")

def controlsesion():

scommon = session # From common

scommonsec = session.secret

spy4web = Session # From py4web

spy4websec = Session.SECRET

return dict(

scommonsec=scommonsec, scommon=scommon, spy4websec=spy4websec, spy4web=spy4web

)

Test template:

[[extend 'layout.html']]

<div>

<h2>Session Secrets from common.py</h2>

<p>scommon [[=scommon]] --- Secret in view: [[=scommon.secret]]</p>

<p>Secret in controller scommonsec [[=scommonsec]]</p>

<hr />

<h2>Session secrets from py4web core.py</h2>

<p>spy4web [[=spy4web]] --- Secret in view: [[=spy4web.SECRET]]</p>

<p>spy4websec [[=spy4websec]]</p>

</div>

Result when running from _dashboard:

Session Secrets from common.py

scommon <py4web.core.Session object at 0x7fac84773430> --- Secret in view: 73a4af6a-9c5a-49b0-9cb5-e716d9b18b60

Secret in controller scommonsec 73a4af6a-9c5a-49b0-9cb5-e716d9b18b60

Session secrets from py4web core.py

spy4web <class 'py4web.core.Session'> --- Secret in view: 5f0cfb1e-5ee2-4731-8a7f-2c4b7440d23a

spy4websec 5f0cfb1e-5ee2-4731-8a7f-2c4b7440d23a

Result when running from main.py:

Session Secrets from common.py

scommon <py4web.core.Session object at 0x7fc8ec4ba220> --- Secret in view: 73a4af6a-9c5a-49b0-9cb5-e716d9b18b60

Secret in controller scommonsec 73a4af6a-9c5a-49b0-9cb5-e716d9b18b60

Session secrets from py4web core.py

spy4web <class 'py4web.core.Session'> --- Secret in view: None

spy4websec None

The None is what drives to the error, but I don't know how to fix it in a secure way.

[Jacinto Parga]

Everything works fine with the new version. I forgot to do: 

    python3 -m pip install -U --no-deps py4web -t lib/

so the corrections in form.py wasn't avaliable.

Now works.

Thanks so much Massimo

[Massimo]

I introduced a vulnerability (unless csrf_session=session is used) Need to rework this.

[Jacinto Parga]

I brief explanation about what I think that happened.

 Let's see the files requirements.txt and Makefile form deployment_tools/gae form source.

The requirements.txt file to deploy in GAE does not include py4web.

So in the Makefile there is the line: python3 -m pip install -U --no-deps py4web -t lib/  this install py4web in the folder lib/ that is deployed to GAE with the rest of the code. So any import from py4web is read from the py4web installed in the lib/ folder

For instance if I upgrade pydal in my developing python environment as it is included in the requirements.txt i will be directly deployed in GAE, but as py4web is not included in the requirements.txt I need to do python3 -m pip install -U --no-deps py4web -t lib/  to upgrade it in the files that are going to be deployed in GAE. Actually I need to include {"path": "lib"} in Visual Code worksapce to properly run it locally.

So whenever an upgrade is made in py4web I need to upgrade it in not only in de python environment but in every GAE app. I don't know why py4web is not in the requirements but I guess it is safer to control what version you have with each application so unwanted changes will not be deployed by chance.

In this case I couldn't make it work until I spread the change in py4web/form.py to lib/ folder.

[Massimo DiPierro]

Now I better understand the question.

You cannot use py4web/requirements.txt to deploy on GAE because GAE fails to install event.

Are you using deployment_tools/gae/?

-- 
You received this message because you are subscribed to the Google Groups "py4web" group.
To unsubscribe from this group and stop receiving emails from it, send an email to py4web+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/py4web/7c270c61-1b3a-4f23-bf96-dfd7dd027339n%40googlegroups.com.

[Jacinto Parga]

Yes, I follow the deployment_tools / gae guidelines with a few small changes

First I create an application  from _dashboard.

Second I create a folder at the same level as apps with the deployment_tools / gae files. From this folder I run the instructions. In the example I call it myappforgae

Finally I move the application that I have created with _dahsboard to the apps directory that I have created in myappforgae folder. I rename the application as _default

It is now ready to be deployed in GAE

I explain it in more detail here https://github.com/jparga/py4web-gae-example