Planning to use appengine/datastore/cloud storage. Have few questions

[Dhanunjaya Naidu Yandrapu]

Hi,

I am new to Google cloud and have few question. I have few questions. I would very much appreciate if you could answer them.

we are planning to use appengine, datastore/cloud storage and golang for my next project.Here are my questions:

1. Currently planning to use appengine but might move to compute engine at a later point of time. is it possible to move from appengine to compute engine and seamlessly access datastore and cloud storage from compute engine(in the same project)?
2. Are there any limits on datastore entity kinds per project?
3. I would also like to know your thoughts on the following design choice for the data model in datastore. We are planning to use name-spaces for multi-tenancy and we have around 100 entity kinds. I am wondering if Approach 2 has any advantages over Approach 1 in terms of query performance or reducing the query cost or deleting all the entities of a particular tenant
• Approach 1: have 100 entity kinds and use name-spaces for tenant isolation. So we always have 100 entity kinds regardless of the number of tenants
  
• Approach 2: use name-spaces for tenant isolation and append tenant ID for each kind, so if we have 10,000 tenants, we will have 1 million entity kinds.

thanks in advance.

 

[Christian F. Howes]

a couple of thoughts...

1. i don't know for sure about datastore connecting to GCE, but the beta managed VMs project allows you to run a GCE instance with full connection to the app engine datastore (and other services).  We have used it well in python for types of processing that we couldn't run on regular app engine instances.
2. i don't know if there are limit on datastore entity kinds per project.
3. i like your approach 1 as it seems much easier to code and use to me.

just my 3 cents...

cfh

[Alex Martelli]

On Sun, Jan 31, 2016 at 5:14 AM, Dhanunjaya Naidu. Yandrapu <dhan...@gmail.com> wrote:

Hi,

I am new to Google cloud and have few question. I have few questions. I would very much appreciate if you could answer them.

we are planning to use appengine, datastore/cloud storage and golang for my next project.Here are my questions:

1. Currently planning to use appengine but might move to compute engine at a later point of time. is it possible to move from appengine to compute engine and seamlessly access datastore and cloud storage from compute engine(in the same project)?

Google Cloud Datastore, https://cloud.google.com/datastore/docs/concepts/overview?hl=en , is accessible from GCE and other platforms. However it's not "seamless" with the very similar feature of App Engine. Cloud Storage, OTOH, *is* "seamless" across all platforms (but has completely different features from a database).

 

1. Are there any limits on datastore entity kinds per project?

The relevant limits is, 200 indices (beyond automatic single-property-ascending ones). If you do any query requiring filtering or ordering on more than one property, that will require an index for each kind you query that way.

 

1. I would also like to know your thoughts on the following design choice for the data model in datastore. We are planning to use name-spaces for multi-tenancy and we have around 100 entity kinds. I am wondering if Approach 2 has any advantages over Approach 1 in terms of query performance or reducing the query cost or deleting all the entities of a particular tenant
• Approach 1: have 100 entity kinds and use name-spaces for tenant isolation. So we always have 100 entity kinds regardless of the number of tenants
  
• Approach 2: use name-spaces for tenant isolation and append tenant ID for each kind, so if we have 10,000 tenants, we will have 1 million entity kinds.

I see no advantage to approach 2. It would surely stop you from using any but the most elementary queries due to the limit on number of indices.

Alex

 

thanks in advance.

 

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/35c69f13-b407-4f6a-b14a-ad1303b49a20%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Wolfram Gürlich]

One thing that makes the datastore stand out is that it doesn't  store meta data about your kinds. Thats's why there isn't a limit to the number of entity kinds or namespaces you can have.

200 kinds is no problem from that perspective. 

Namespaces make your life much easier. Your 2nd approach is pretty much what appengine does on the inside when you use namespaces.

The limit on custom indexes is a serious problem however. Our solution was to not use them at all. We use single indexed properties instead and manage our composite indexes on top of that. That's much less pain than it may sound at first.

To become a happy datastore user you must overcome relational thinking. There's more to it than to just stop thinking in terms of "tables". You should design for a more coarse-grained data scheme. In general that means identifying the entities that will almost always be used together - like an order and its order items. Store them as closely together as you can. Entity groups are a good start but also take a look at embedded entities.

What we did is to store all business data as JSON Blobs inside entities and use regular properties for the *sole* purpose of indexing. That has been working out *very* well for us.

Regards,

Wolfram

[Dhanunjaya Naidu Yandrapu]

Thanks Wolfram for sharing your thoughts and i very much appreciate it.

I was also thinking of storing business data as a blob and using indexes only to select that data. Thanks for confirming that this approach is working very well.

I have one more question regarding namespaces and entity groupings. Lets assume 

1. entity1, entity2 are in namespace ns1 
2. entity3, entity4 are in namespace ns2
3. entity1, entity3 has parent p1 
4. entity2, entity4 has parent p2
   

are entities 1 and 3 in different groups from a transaction perspective because they belong to different namespaces?

thanks,

dhanunjay

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/H03vcmqrkZw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.

To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/de93c24c-f2eb-4300-9104-8c17f6b10459%40googlegroups.com.

[PK]

I think the answer is yes. Look at this: http://stackoverflow.com/questions/10306753/entity-groups-and-namespaces

--Panos

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.

To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAHwUyAr7nNdOfroZcV_UEEptUTRNQ5aYCMQEApoEiwThJh%3D2uQ%40mail.gmail.com.