Structured vs. Key Properties

[Katy Lion]

Hi!

We are using ndb to model our project. We have a model that looks something similar to this:

class Model1
    attribute1 = ndb.StringProperty()

class Model2
   attribute2 = = ndb.LocalStructuredProperty(Model1, repeated=True)

class Model3
   attribute3 = = ndb.LocalStructuredProperty(Model2, repeated=True)

class Model4
   attribute4 = = ndb.LocalStructuredProperty(Model3, repeated=True)


Since we have repeated attributes in each model, we can only use LocalStructuredProperty. These LocalStructuredProperty contain global information that could be updated at any point. However, updating information is a hassle because the LocalStructuredProperties are not indexed. 

So, we switched our design to use KeyProperties:

class Model1
   attribute1 = ndb.StringProperty()

class Model2
   attribute2 = = ndb.KeyProperty(kind=Model1, repeated=True)

class Model3
   attribute3 = = ndb.KeyProperty(kind=Model2, repeated=True)

class Model4
   attribute4 = = ndb.KeyProperty(kind=Model3, repeated=True)

Since we have to get the values of each key, this design slows down the reads. 

We created a single page app in frontend and we want to send a json that returns the values as follows:

[

    {

        "attribute4": [

            {

                "attribute3": [

                    {

                        "attribute2": [

                            {

                                "attribute1": "somevalue"

                            }

                        ]

                    }

                ]

            }

        ]

    }

]

Iterating, reading and returning the values of 20 Model4's, takes at least 5.5 seconds. This worries us because  it takes this much time even though we are trying to get a relatively small amount of data. 

The advantage of of using KeyProperty is that updating information becomes easy, but the reads are slower. But we can't switch back to StructuredProperty because the updates are too complicated. We tried to save the references but we would have to iterate through each Model to get the value that we want to update. We are considering changing to CloudSQL, instead of ndb. What approach should we choose? What suggestions do you have?

I would really appreciate your advice!

[Peter Retief]

I did something similar, my data set was quite small though, was quite speedy - certainly faster than 5 seconds

Bit more information might help?

btw I am combining fuill text search with models and seems to be working ok, just a thought, don't shout 

https://cloud.google.com/appengine/docs/python/search/

class Line(ndb.Model):

    created = ndb.DateTimeProperty(auto_now_add=True)

    updated = ndb.DateTimeProperty(auto_now=True) 

    name = ndb.StringProperty(default="Message") 

    text = ndb.StringProperty(required=True)

    status = ndb.StringProperty(default="0") 

class Message(ndb.Model):

    created = ndb.DateTimeProperty(auto_now_add=True)

    updated = ndb.DateTimeProperty(auto_now=True) 

    name = ndb.StringProperty(default="Message group") 

    duration = ndb.StringProperty(default="2") 

    effect = ndb.StringProperty(default="Auto") 

    font = ndb.StringProperty(default="bold") 

    line =  ndb.KeyProperty(kind=Line, repeated=True) 

    status = ndb.StringProperty(default="0")

class Device(ndb.Model):

    created = ndb.DateTimeProperty(auto_now_add=True)

    updated = ndb.DateTimeProperty(auto_now=True) 

    name = ndb.StringProperty(required=True) 

    message = ndb.KeyProperty(kind=Message, repeated=True)

    status = ndb.StringProperty(default="0")

class User(User):

    device = ndb.KeyProperty(kind=Device, repeated=True)

    created = ndb.DateTimeProperty(auto_now_add=True)

    updated = ndb.DateTimeProperty(auto_now=True)

    username = ndb.StringProperty(required=True)

    name = ndb.StringProperty()

    last_name = ndb.StringProperty()

    email = ndb.StringProperty(required=True)

--
You received this message because you are subscribed to the Google Groups "appengine-ndb-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to appengine-ndb-di...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Katy Lion]

A list of one Model4 looks like this:

[
    {
        "attribute4": [
            {
                "attribute3": [
                    {
                        "attribute2": [
                            {
                                "attribute1": "somevalue"

                            },
                            {
                                "attribute1": "somevalue"
                            },
                            {
                                "attribute1": "somevalue"
                            }
                        ]
                    },
                    {
                        "attribute2": [
                            {
                                "attribute1": "somevalue"
                            },
                            {
                                "attribute1": "somevalue"
                            },
                            {
                                "attribute1": "somevalue"
                            }
                        ]
                    },
                    {
                        "attribute2": [
                            {
                                "attribute1": "somevalue"
                            },
                            {
                                "attribute1": "somevalue"
                            },
                            {
                                "attribute1": "somevalue"
                            }
                        ]
                    }
                ]
            },
            {
                "attribute3": [
                    {
                        "attribute2": [
                            {
                                "attribute1": "somevalue"
                            },
                            {
                                "attribute1": "somevalue"
                            },
                            {
                                "attribute1": "somevalue"
                            }
                        ]
                    },
                    {
                        "attribute2": [
                            {
                                "attribute1": "somevalue"
                            },
                            {
                                "attribute1": "somevalue"
                            },
                            {
                                "attribute1": "somevalue"
                            }
                        ]
                    },
                    {
                        "attribute2": [
                            {
                                "attribute1": "somevalue"
                            },
                            {
                                "attribute1": "somevalue"
                            },
                            {
                                "attribute1": "somevalue"
                            }
                        ]
                    }
                ]
            },
            {
                "attribute3": [
                    {
                        "attribute2": [
                            {
                                "attribute1": "somevalue"
                            },
                            {
                                "attribute1": "somevalue"
                            },
                            {
                                "attribute1": "somevalue"
                            }
                        ]
                    },
                    {
                        "attribute2": [
                            {
                                "attribute1": "somevalue"
                            },
                            {
                                "attribute1": "somevalue"
                            },
                            {
                                "attribute1": "somevalue"
                            }
                        ]
                    },
                    {
                        "attribute2": [
                            {
                                "attribute1": "somevalue"
                            },
                            {
                                "attribute1": "somevalue"
                            },
                            {
                                "attribute1": "somevalue"
                            }
                        ]
                    }
                ]
            }
        ]
    }
]
I realized that I was using a terrible method to retrieve the data. I got all the Model4, iterated on the list of Model4's, got all the Model3's in each Model4 and repeated the same process for each level. This is how I got the 5.5 seconds.

I tried to optimize the method but the lowest time I was able to get was 3.5 seconds. I get all the Model4's, iterate on the list of Model4's, got all the keys that I need of Model3, get the Model3's with the list of keys, assign the Model3 to the correct object of Model4 and repeat the same process for each level.

I hope this is not that confusing... I am aware that my method is not that great but I am open for suggestions if you have any idea of how to improve it.

I actually think that indexing the models is a very interesting idea. Using your example, how exactly should I index it? Should an User and its attributes (Device, Messages and Lines) be under the same index name? Should the index contain only keys or should it contain a replica of the objects? Does this mean that when I update the data, I will also have to update the index?