Datastore available to py4web

[Jacinto Parga]

Hi,
Pydal works well with SQL databases in Google Appengine, but google:datastore and google:datastore+nbd don't work with python3.

I have tried to migrate the GoogleDatastore(NoSQLAdapter) with no success till now.

It's been published new guides to move from the deprecaed google libraries:

https://github.com/GoogleCloudPlatform/appengine-python2-3-migration/tree/main/datastore

https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-2-cloudndb?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudndb_201021&utm_content=-#0

https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-3-datastore#0

https://github.com/googlecodelabs/migrate-python2-appengine

I'm asking for some help to upgrade this pydal feature to make it avaliable to py4web.

[Massimo DiPierro]

I started working on this. Is there an easy way to test this locally without deploying to GAE?

Anybody willing to help with testing?

--
You received this message because you are subscribed to the Google Groups "py4web" group.
To unsubscribe from this group and stop receiving emails from it, send an email to py4web+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/py4web/d7477502-b51d-4f84-ad49-b1f96159af70n%40googlegroups.com.

[Jacinto Parga]

Hi, 

I can test it locally and deploying it in GAE if necessary. :)

Thanks Massimo

[Massimo]

I made this PR for pydal and it works with the posted example (using local datastore).

https://github.com/web2py/pydal/pull/682

I could definitely use some help testing. In particular I did not try more complex field types (blob, upload, reference, list:string, etc) and/or complex queries.

OR and NOT do not work on the new datastore as far as I understand so they are no longer supported.

I added support for projections but not for orderby and limitby.

Massimo

[Jacinto Parga]

Hi,

 thanks Massimo. I have started testing it. Comments in github.

[Massimo]

This helped a lot.

I added support for limitby and orderby. I added more debug info.

I can only run from a normal python program. When running in a py4web app, it does not connect to the datastore. Do not know why. Hence I cannot reproduce and fix the problem with auth. I added more debug info so maybe you can help me figure out which validator creates the problem.

Massimo

[Massimo]

Moved all the legacy ndb logic to ndp.py files SQL("google:datastore+ndb") and all the new datastore logic to DAL("google:datastore").

The latter is not backwards compatible and requires integer Key ids.

I may have fixed the problem with Auth.

I have tested whether NDB works or is broken.

Massimo

[Jacinto Parga]

Great. I think it is the best decission.

I have made new tests. 

Auth still breaks on register, 

Traceback (most recent call last): File "/srv/lib/py4web/core.py", line 916, in wrapper ret = func(*func_args, **func_kwargs) File "/srv/lib/py4web/core.py", line 867, in wrapper ret = func(*args, **kwargs) File "/srv/lib/py4web/core.py", line 867, in wrapper ret = func(*args, **kwargs) File "/srv/lib/py4web/utils/auth.py", line 918, in _ form=form_factory(), path=path, user=auth.get_user(), **env File "/srv/lib/py4web/utils/auth.py", line 1174, in register res = self.auth.register(form.vars, validate=False) File "/srv/lib/py4web/utils/auth.py", line 511, in register res = store(fields) File "/srv/lib/py4web/utils/auth.py", line 504, in store return dict(id=self.db.auth_user.insert(**fields)) File "/srv/lib/pydal/objects.py", line 892, in insert ret = self._db._adapter.insert(self, row.op_values()) File "/srv/lib/pydal/adapters/google.py", line 282, in insert self.client.put(entity) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/datastore/client.py", line 575, in put self.put_multi(entities=[entity], retry=retry, timeout=timeout) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/datastore/client.py", line 612, in put_multi current.put(entity) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/datastore/batch.py", line 227, in put _assign_entity_to_pb(entity_pb, entity) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/datastore/batch.py", line 373, in _assign_entity_to_pb bare_entity_pb = helpers.entity_to_protobuf(entity) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/datastore/helpers.py", line 215, in entity_to_protobuf _set_protobuf_value(value_pb, value) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/datastore/helpers.py", line 449, in _set_protobuf_value attr, val = _pb_attr_value(val) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/datastore/helpers.py", line 366, in _pb_attr_value raise ValueError("Unknown protobuf attr type", type(val)) ValueError: ('Unknown protobuf attr type', <class 'pydal.validators.LazyCrypt'>)

but I declared some indexes:

- kind: auth_user

properties:

- name: email

- name: id

- kind: auth_user

properties:

- name: username

- name: id

And then I could create one user from the admin grid I had implemented.

Request_password works and with the right configuration (gae configuration fails as I explain later), sends the mail to change the password, but reset_password fails because the like _or expresions:

"Traceback (most recent call last): File "/srv/lib/py4web/core.py", line 916, in wrapper ret = func(*func_args, **func_kwargs) File "/srv/lib/py4web/core.py", line 867, in wrapper ret = func(*args, **kwargs) File "/srv/lib/py4web/core.py", line 867, in wrapper ret = func(*args, **kwargs) File "/srv/lib/py4web/utils/auth.py", line 918, in _ form=form_factory(), path=path, user=auth.get_user(), **env File "/srv/lib/py4web/utils/auth.py", line 1323, in reset_password user = self.auth.db(query).select().first() File "/srv/lib/pydal/objects.py", line 2719, in select return adapter.select(self.query, fields, attributes) File "/srv/lib/pydal/adapters/google.py", line 235, in select items, table = self.select_raw(query, fields, attributes) File "/srv/lib/pydal/adapters/google.py", line 195, in select_raw for item in self.parse_and_query(query): File "/srv/lib/pydal/adapters/google.py", line 158, in parse_and_query raise RuntimeError("Datastore cannot handle expressions like %s" % query.op.__name__) RuntimeError: Datastore cannot handle expressions like _or"

The mail() has also errors in gae due to the use of the google.appengine.api library:

try:

from google.appengine.api import mail as google_mail

GAE = True

except ImportError:

GAE = False

and this is the error trace:

File "/srv/lib/py4web/utils/auth.py", line 804, in send return self.sender.send(email, subject=subject, body=body) File "/srv/lib/py4web/utils/mailer.py", line 758, in send raise RuntimeError("Not running on GAE") RuntimeError: Not running on GAE

Finally I tried a grid for the auth_user table and I got this error:

Traceback (most recent call last): File "/srv/lib/py4web/core.py", line 916, in wrapper ret = func(*func_args, **func_kwargs) File "/srv/lib/py4web/core.py", line 867, in wrapper ret = func(*args, **kwargs) File "/srv/apps/best/controllers.py", line 93, in authgrid grid = Grid( File "/srv/lib/py4web/utils/grid.py", line 447, in __init__ self.process() File "/srv/lib/py4web/utils/grid.py", line 655, in process self.rows = db(query).select(*self.needed_fields, **select_params) File "/srv/lib/pydal/objects.py", line 2719, in select return adapter.select(self.query, fields, attributes) File "/srv/lib/pydal/adapters/google.py", line 235, in select items, table = self.select_raw(query, fields, attributes) File "/srv/lib/pydal/adapters/google.py", line 217, in select_raw first, op, second = item.op(item.first, item.second) AttributeError: 'list' object has no attribute 'op'

[Massimo DiPierro]

Thank you. Will take another pass at it.

To view this discussion on the web visit https://groups.google.com/d/msgid/py4web/8f9c5f6b-591c-4052-bf6d-66d59228bfa5n%40googlegroups.com.

[Jacinto Parga]

The new version of py4web   1.20220928.1 fails wiht Google Datastore adapter.

This is the error:

Traceback (most recent call last): 

 File "/srv/lib/py4web/core.py", line 928, in wrapper ret = func(*func_args, **func_kwargs) 

 File "/srv/lib/py4web/core.py", line 913, in wrapper raise context["exception"] 

 File "/srv/lib/py4web/core.py", line 890, in wrapper call(fixture.on_request, context) 

 File "/srv/lib/py4web/core.py", line 875, in call return f(context) 

 File "/srv/lib/py4web/core.py", line 371, in on_request self.get_connection_from_pool_or_new() 

 File "/srv/lib/pydal/base.py", line 790, in __getattr__ return BasicStorage.__getattribute__(self, key) 

AttributeError: 'DAL' object has no attribute 'get_connection_from_pool_or_new'

It is caused by the commit

https://github.com/web2py/py4web/commit/3401f153279bba7e87739806fb6df45f5587f723#diff-a6dba174e82017ed342c8bba4aaa3e886be2b97449d6a3578c7c9a5fda50f516

I've been doing several test wiht Datastore Adapter "google:datastore" and it works quite well (not perfect) with the py4web version 1.20220615.1

There are problems with auth.py and grid.py due to the db queries not supported by Datastore. Some can be solved with an intensive index declaration. Anyway they can be solved with simple queries and Datatables for the most common situations.

I would like to test if it works wiht ombott instead of bottle in main.py but this error has to be fixed first.