GAE db datastore connection

[Jacinto Parga]

Hi, 

I have deployed an app in GAE and works fine with these  db settings in settings.py:

# db settings

APP_FOLDER = os.path.dirname(__file__)

APP_NAME = os.path.split(APP_FOLDER)[-1]

# DB_FOLDER:    Sets the place where migration files will be created

#               and is the store location for SQLite databases

DB_FOLDER = os.path.join(APP_FOLDER, "databases")

DB_URI = "sqlite://storage.db"

DB_POOL_SIZE = 1

But it's not connected to google:datastore. If I just change:

# db settings

APP_FOLDER = os.path.dirname(__file__)

APP_NAME = os.path.split(APP_FOLDER)[-1]

# DB_FOLDER:    Sets the place where migration files will be created

#               and is the store location for SQLite databases

DB_FOLDER = os.path.join(APP_FOLDER, "databases")

DB_URI = "google:datastore"

DB_POOL_SIZE = 1

It returns a not found error.

How can I configure py4web db connection to datastore?

Thanks

[Jacinto Parga]

The last configuration I tried:

# db settings

APP_FOLDER = os.path.dirname(__file__)

APP_NAME = os.path.split(APP_FOLDER)[-1]

# DB_FOLDER:    Sets the place where migration files will be created

#               and is the store location for SQLite databases

DB_FOLDER = None # os.path.join(APP_FOLDER, "databases")

DB_URI = "google:datastore" # "sqlite://storage.sqlite"

DB_POOL_SIZE = 0 # 1

Got this error:

 raise ValueError("no %s found for object: %s" % (self.namespace, obj))
ValueError: no dialect found for object: <pydal.adapters.google.GoogleDatastore object at 0x7f30dc76c5d0>

Still don't know how to connnect the applictaion to datastore. Someone knows to configure it?

Thanks

[Massimo]

Unfortunately at this time py4web does not work with the datastore because google app engine for python 3 only supports new APIs and pydal only supports the old APIs. :-(

This needs to be fixed.

[Jacinto Parga]

I see, thanks for the answer. So I guess there is a similar issue with web2py because uses pydal.

[Massimo DiPierro]

Only if you use py3 on gae. Web2py works well for py2 in gae

--
You received this message because you are subscribed to the Google Groups "py4web" group.
To unsubscribe from this group and stop receiving emails from it, send an email to py4web+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/py4web/5e8af744-1932-4f26-ac01-2de6d3e0b58a%40googlegroups.com.

[Tolga Konik]

py3 support is much better in gae compared to py2. In the standard setup, google allows installs with py3 while it does not for py2.

Looked into this a little. In pydal _gae file, there is reference to "google.appengine", which last time I checked was not supported in py3. Google now has an alternative library with a similar API for the datastore:

https://pypi.org/project/google-cloud-datastore/

which I believe has similar functionality, e.g.,

        from google.cloud import ndb

        from google.cloud.ndb.polymodel import PolyModel as NDBPolyModel

To unsubscribe from this group and stop receiving emails from it, send an email to py4...@googlegroups.com.

[Tolga Konik]

The datastore library below is the current recommended but it has slightly different API. The following looks more compatible with what is already implemented with pydal:

google-cloud-ndb

https://pypi.org/project/google-cloud-ndb/

[Tolga Konik]

following up:

Here is a good article migrating the datastore ndb interface from python2 to python3. 

https://cloud.google.com/appengine/docs/standard/python3/migrating-to-cloud-ndb

I got a rudimentary version of it working, but probably needs cleanup and its coverage may be limited. With these changes, Auth(db) worked for me with google:datastore, using local data store emulator. Some more details are on pull request description, but I will be happy to answer questions.

https://github.com/web2py/pydal/pull/624

[Jacinto Parga]

Great job , close to solution.

Is pydal the only library to modify to achieve the compatibility? I suposse the changes required in pydal to make datastore work with py4web won’t be enough to make web2py work too.

[Massimo DiPierro]

Pydal is the only problem. It uses gae API that are deprecated. There is working under way to fox this within one week. Will you be able to help with tests?

On Fri, Apr 24, 2020, 07:43 Jacinto Parga <jpa...@gmail.com> wrote:

Great job , close to solution.

Is pydal the only library to modify to achieve the compatibility? I suposse the changes required in pydal to make datastore work with py4web won’t be enough to make web2py work too.

--
You received this message because you are subscribed to the Google Groups "py4web" group.

To unsubscribe from this group and stop receiving emails from it, send an email to py4web+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/py4web/57240fd1-d3e0-4be7-82e5-65aa8138616b%40googlegroups.com.

[Jacinto Parga]

Yes, I will test it.

Startpoint https://web2gae.appspot.com/
http://py4websqlite.web2gae.appspot.com/wiki

If we use sqlite as db py4web apps can be deployed in GAE. If we upload database files the records created in localhost can be readed, but you cannot write new records.

Will continue

[Jacinto Parga]

I have tried to deploy in gae web2py 2.20.4 with python 2.7.17

Again an error related to pydal raises in login:

2020-05-11 19:11:48.041 CEST Unable to store in FILE: Framework (/base/data/home/apps/s~web2gae/1.426602206860768093/gluon/restricted.py:66)

Traceback (most recent call last):
  File "/base/data/home/apps/s~web2gae/1.426602206860768093/gluon/main.py", line 456, in wsgibase
    session._try_store_in_db(request, response)
  File "/base/data/home/apps/s~web2gae/1.426602206860768093/gluon/globals.py", line 1251, in _try_store_in_db
    if not table._db(table.id == record_id).update(**dd):
  File "/base/data/home/apps/s~web2gae/1.426602206860768093/gluon/packages/dal/pydal/objects.py", line 2686, in update
    ret = db._adapter.update(table, self.query, row.op_values())
  File "/base/data/home/apps/s~web2gae/1.426602206860768093/gluon/packages/dal/pydal/adapters/google.py", line 440, in update
    setattr(item, field.name, self.represent(value, field.type))
  File "/base/data/home/apps/s~web2gae/1.426602206860768093/gluon/packages/dal/pydal/adapters/google.py", line 248, in represent
    return super(GoogleDatastore, self).represent(obj, field_type)
  File "/base/data/home/apps/s~web2gae/1.426602206860768093/gluon/packages/dal/pydal/adapters/base.py", line 384, in represent
    return self.representer.represent(obj, field_type)
  File "/base/data/home/apps/s~web2gae/1.426602206860768093/gluon/packages/dal/pydal/representers/__init__.py", line 246, in represent
    rv = self.get_representer_for_type(field_type)(rv, field_type)
  File "/base/data/home/apps/s~web2gae/1.426602206860768093/gluon/packages/dal/pydal/representers/__init__.py", line 138, in __call__
    return self.adapt(self.call(value, field_type))
  File "/base/data/home/apps/s~web2gae/1.426602206860768093/gluon/packages/dal/pydal/representers/__init__.py", line 135, in _call
    return self.inner_call(value)
  File "/base/data/home/apps/s~web2gae/1.426602206860768093/gluon/packages/dal/pydal/representers/__init__.py", line 123, in _inner_call
    return self.obj.f(self.representer, value, **kwargs)
  File "/base/data/home/apps/s~web2gae/1.426602206860768093/gluon/packages/dal/pydal/representers/base.py", line 267, in _datetime
    time_items = list(map(int, str(value)[11:].strip().split(":")[:3]))
ValueError: invalid literal for int() with base 10: '47.414717'

The last web2py version that works without problem in gae is 2.18.5

I have also tried to use pydal from Tolga Konik (https://github.com/web2py/pydal/pull/624), but I could not configure it for py4web. How can I try that pydal with py4web?

El viernes, 24 de abril de 2020, 16:50:44 (UTC+2), Massimo escribió:

Pydal is the only problem. It uses gae API that are deprecated. There is working under way to fox this within one week. Will you be able to help with tests?

On Fri, Apr 24, 2020, 07:43 Jacinto Parga <jpa...@gmail.com> wrote:

Great job , close to solution.

Is pydal the only library to modify to achieve the compatibility? I suposse the changes required in pydal to make datastore work with py4web won’t be enough to make web2py work too.

--
You received this message because you are subscribed to the Google Groups "py4web" group.

To unsubscribe from this group and stop receiving emails from it, send an email to py4...@googlegroups.com.

[Massimo]

latest pydal should fix this. Can you help check it?

[Jacinto Parga]

Hi, 

yes latest pydal 20200524.3 fix it.

Deployment in GAE with web2py Version 2.20.4-stable+timestamp.2020.05.03.05.18.50 allows to write and read in Datastore https://web2gae.appspot.com/

I only had an issue with the embebed wiki that does noT allow to update the values or to create a new one, but it allows to import them from a csv file, so I think the issue has not to do with pydal but with some code mistake.

To do:

• python 3 deployments with py4web in GAE using Datastore db "google:datastore"
• python 3 web2py deployments in GAE

It is important to solve at least one of this challenges as it is necessary to migrate to python 3 in the web2py apps deployed in GAE.

Thanks Massimo

[Massimo DiPierro]

At this time you cannot. We are working on it.

--

You received this message because you are subscribed to the Google Groups "py4web" group.

To unsubscribe from this group and stop receiving emails from it, send an email to py4web+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/py4web/70197c01-7538-41fb-a56b-675ecfc9e7b7%40googlegroups.com.

[Jacinto Parga]

Hello @Massimo, 

there has been a lot of improvements in py4web. 

I am planning to start a new project, and I wonder if it would be possible to use Datastore in pydal with py4web. I have made a workaroud with the client library (https://googleapis.dev/python/datastore/latest/client.html) in the tests, but there is a lot of features in pydal that are worthwhile.

Do you think it is out of focus today? 

Thanks