Datastore field formats

64 views
Skip to first unread message

Christopher Bourez

unread,
Jul 27, 2016, 10:58:59 AM7/27/16
to Google App Engine
I've been trying to save some data in the datastore following the google Bookshelf python example, with the gcloud python library, as explained in the tutorial. 

It was working well to save small strings and times. 

The problem is for other formats. For example, if I want to save a big JSON : 

    data['result'] = json.dumps(tags[0])
    ds = datastore.Client(current_app.config['PROJECT_ID'])
    if id:
        key = ds.key('Pred', id)
    else:
        key = ds.key('Pred')

    entity = datastore.Entity(
        key=key,
        exclude_from_indexes=['description'])

    entity.update(data)
    ds.put(entity)
    return from_datastore(entity)

It says that the string is too long (more than 1500) to be accepted. 

I've been trying other solutions, such as using the google appengine ndb library, but when using this library from a computer outside the compute engine, it said there was missing a proxy api for the datastore v3.

Sounds also the example in the tutorial follows DB pratices, and not NDB.

The same question as the question about how to insert a text format field in the datastore, is true for other formats.

Thanks

Clement Nodet

unread,
Jul 27, 2016, 12:01:18 PM7/27/16
to Google App Engine
Hi Christopher,

in Datastore, long strings (larger than 1500 bytes) cannot be indexed (https://cloud.google.com/datastore/docs/concepts/entities#text_string)
The issue here is that, by default, Datastore tries to index every property you give it, unless told otherwise.

Here, it tries to index the property "result", which contains the large JSON string.

You can tell it to explicitly not index that property with:

    entity = datastore.Entity(
        key=key,
        exclude_from_indexes=['description', 'result'])

Hope that helps!

Vitaly Bogomolov

unread,
Jul 27, 2016, 3:39:37 PM7/27/16
to Google App Engine
Hi, All.



in Datastore, long strings (larger than 1500 bytes) cannot be indexed (https://cloud.google.com/datastore/docs/concepts/entities#text_string)
The issue here is that, by default, Datastore tries to index every property you give it, unless told otherwise.

Here, it tries to index the property "result", which contains the large JSON string.

You can tell it to explicitly not index that property with:

    entity = datastore.Entity(
        key=key,
        exclude_from_indexes=['description', 'result'])

Hope that helps!

or use db.TextProperty. its size up to 1Mb, and not indexed too.

Christopher Bourez

unread,
Aug 2, 2016, 4:58:48 AM8/2/16
to Google App Engine
great Clement!

@Vitaly : how can I use db.TextProperty in the example I gave ? because it was not obvious for me, i've tried a lot

Vitaly Bogomolov

unread,
Aug 2, 2016, 3:33:48 PM8/2/16
to Google App Engine

@Vitaly : how can I use db.TextProperty in the example I gave ? because it was not obvious for me, i've tried a lot

Vitaly Bogomolov

unread,
Aug 2, 2016, 3:45:19 PM8/2/16
to Google App Engine

@Vitaly : how can I use db.TextProperty in the example I gave ? because it was not obvious for me, i've tried a lot

 

if your json is really BIG, maybe you need zip it and save to datastore BlobProperty or use Cloud Storage.

Reply all
Reply to author
Forward
0 new messages