Finishing of READ and WRITE operation

89 views
Skip to first unread message

Anirban Silsarma

unread,
Sep 21, 2015, 3:23:36 AM9/21/15
to Google App Engine
How can i minimize read write operation, and how can I understand how much read and write operation has occurred in per transaction?? 

Patrice (Cloud Platform Support)

unread,
Sep 21, 2015, 11:05:37 AM9/21/15
to Google App Engine
Hi Anirban,

While questions about best practices, optimization, and the like, are on topic for Google Groups, we need a LOT more context into your application to be able to provide ANY kind of answer here. At the time of reading, I'm not even sure if you're talking Datastore or CloudSQL. In any case, not knowing what models you have, what your application does, where do you want lower latency, what's done backend or frontend, this is literally impossible to answer.

I can give generic tips for now, like to use "getByKeys", since they don't cost reads. You could also make use of memcache for non-critical data that is updated enough (but memcache can flush before you end up saving it, so make sure this IS non-critical). But that's assuming you're discussing Cloud Datastore (in which case I would suggest you give this article).

In any case, provide a bit more context into your question, and either myself or someone from the community will be happy to help.

Cheers!

Anastasios Hatzis

unread,
Sep 21, 2015, 3:21:28 PM9/21/15
to google-a...@googlegroups.com
You didn't mention which language or API you use. But I assume you are referring to Google Cloud Datastore in general. Here some tricks I've learnt the hard way, and sometimes the easy way by watching Google Developers videos and blog posts. ;-) I hope they help.

Minimize read operations in Google Cloud Datastore:
Try to have larger entities, rather than many small entities with references. What makes sense in SQL (e.g. 5th Normal Form) does make not much sense in a non-SQL tech. Especially because you don't have joint queries and such.

Start with the most popular request handlers (that you expect in case of a new app or according to analytics). What information will they need most? Could you put them into the same entity kind? Can you get the entities by key or do you need queries? By key is faster. And by key would also allow easier utilisation of memcache in some use-cases.

Try to avoid round-trips, i.e. first get some entities, then depending on property values in the results read more entities from datastore. Better: Try to get and query everything in one (parallel) step. Have the client directly provide keys/IDs or whatever you need (but of course, do never trust the client!). Much faster and cheaper.

For example, a SalesOrderItem query would need not only Product ID, Description, Quantity, and Price, but you also want to show Customer ID, Date of Purchase, and Sales Order ID, and a thumbnail of the primary product photo.

Now, in SQL you would have SalesOrderItem table with key columns referencing to a Customer table, and a SalesOrder table, then doing a joint query. If you try hard to impose SQL thinking you might see this workaround: first query the items, then for each of them the referenced product, sales order, and customer.

In Google Cloud Datastore though, it makes often more sense to store every important information of a query directly in the entity kind, especially the properties that are unlikely to change afterwards.

Example for SalesOrderItem kind:
  • parent (key to SalesOrder entity, key contains ID of the SalesOrder entity, too)
  • quantity (float)
  • price (float)
  • productID (int)
  • dateOfPurchase (date copied from SalesOrder, should never change anyway)
  • customer (key, but key contains the unique ID of customer account)
  • productDescr (str, copy the description property from the Product entity; if a description is changed, you probably don't need to propagate it anyways to old sales order items, because when the sales order is created the sold products are "as is")
  • productPhoto (URL-safe encoded key of the product; store all product thumbnails in e.g. Google Cloud Storage and use that string in the path, so you can automatically build the image source path)
This way, you can query and show sales order items in one step, with-out additionally reading sales-orders, customers, and products. This translates to 1 read per item rather than 4 or more reads.

Use keys-only queries if you only care for count of results. Use fetch-page with cursors, because with offset instead, the datastore will really go through each entity in the query until it reaches the desired offset.

Eventually, you can use projected queries. I've never used them, but basically, you define in the read op which property values you need for a query. Those are billed as "small ops". I'm not sure if projected queries are supported in all APIs.

Minimize write operations:
Use shorter property names for less storage. Especially important for high volume kinds.

Only index the properties you definitely need for ordering or filtering of queries. You can index properties later if the app really needs some new query filter or so. 

The more indexed properties are defined for entity kind, the more write ops with every new entity and entity update (exponentially!).

Also important aspect regarding write operations is scalability - Datastore is all about scalability, but the model needs to pay respect to some technical implications: Try to model in a way that makes sure that there will be no excessive write ops into the same entity group. Entity groups allows your app strong consistency with-in the entity group, but Datastore has a write limit of 1 write per second per entity group. For example, SalesOrder and SalesOrderItem could be in the same entity group, if it is unlikely that more than one user will work on the same sales-order at the same time. If you don't update individual items but the sales order and all its items together in single requests, it is less likely you will hit the limit. Probably bad practice would be to put all sales-orders also into the same entity group, e.g. all of them have OrderBook as parent, your app could very likely hit the limit frequently.

Some more reading:

Regarding the number of read/write ops inside a transaction, I don't know if these numbers are available.

2015-09-21 9:23 GMT+02:00 Anirban Silsarma <anirban....@infiflex.co>:
How can i minimize read write operation, and how can I understand how much read and write operation has occurred in per transaction?? 

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/2ffcfc73-5bf3-4d03-b783-62b557dc81b2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--

Mit freundlichen Grüßen / Kind regards

i. A.
Anastasios Hatzis

Fon: +49 8374 930813
Fax: +49 8374 930810
Mobil: +49 1520 8592878

HATZIS Edelstahlbearbeitung GmbH
Hojen 2
87490 Haldenwang (Allgäu)
Germany

Handelsregister Kempten (Allgäu): HRB 4204
Geschäftsführer: Paulos Hatzis, Charalampos Hatzis
Umsatzsteuer-Identifikationsnummer: DE 128791802
GLN: 42 504331 0000 6

http://www.hatzis.de/

Patrice (Cloud Platform Support)

unread,
Sep 22, 2015, 9:46:08 AM9/22/15
to Google App Engine
Hi Anastasios,

Thank you VERY much for that detailed explanation. This is great and exactly what we wanted Groups to be when we set it up :). I have a feeling I may use your post as a reference for other customers :).

Keep up the amazing work. Cheers!
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.



--

Mit freundlichen Grüßen / Kind regards

i. A.
Anastasios Hatzis

Fon: +49 8374 930813
Fax: +49 8374 930810
Mobil: +49 1520 8592878

Anastasios Hatzis

unread,
Sep 22, 2015, 1:55:51 PM9/22/15
to Google App Engine
Hi Patrice!

Many thanks... your appreciation is much appreciated ;-) I just hope, the original poster found it useful, too.  However, you are free to use my post in any way you think is suitable.

Cheers
Ani
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.



--

Mit freundlichen Grüßen / Kind regards

i. A.
Anastasios Hatzis

Fon: +49 8374 930813
Fax: +49 8374 930810
Mobil: +49 1520 8592878
Reply all
Reply to author
Forward
0 new messages