How to think about the Datastore
48 views
Skip to first unread message
Magnus Gunnarsson
Jan 27, 2012, 12:28:09 PM1/27/12
to google-a...@googlegroups.com
I have been experimenting a bit with the Datastore and it seems to be a good thing with many possibilities. However, I am uncertain about how it is intended to be used, what the typical design patterns should look like. I am used to relational databases and know their strengths and weaknesses, and I design my applications accordingliy. I suspect that is somewhat inadequate when using GAE.
I have a sign language dictionary (8000 entries) where I fetch data from an external server, store it in GAE and present it to the user (in a hopefully appealing way) using GWT. My current design (which is not yet fully implemented) is to store each dictionary entry as an Entity in the Datastore, with most of the information pieces for the entry as unindexed properties of the Entity. The searches in the dictionary are sometimes quite complex, and the search capabilities in Datastore are not sophisticated enough. Thus I create a small number of indexes and put these in MemCache.
Now, it seems that the application hit the quota limitations quite easily. Just loading the dictionary from the external server seems to be impossible with out hitting the Datastore write limit. (For the free version, of course. I have no money.)
So, I am considering serialising each dictionary entry and storing them as Blobs (or Texts), individually. Another option is to serialise the entire dictionary and store it in a Blob. (I do that, in practice, in the old version of the dictionary application, written in PHP.)
Any advice?
I have a sign language dictionary (8000 entries) where I fetch data from an external server, store it in GAE and present it to the user (in a hopefully appealing way) using GWT. My current design (which is not yet fully implemented) is to store each dictionary entry as an Entity in the Datastore, with most of the information pieces for the entry as unindexed properties of the Entity. The searches in the dictionary are sometimes quite complex, and the search capabilities in Datastore are not sophisticated enough. Thus I create a small number of indexes and put these in MemCache.
Now, it seems that the application hit the quota limitations quite easily. Just loading the dictionary from the external server seems to be impossible with out hitting the Datastore write limit. (For the free version, of course. I have no money.)
So, I am considering serialising each dictionary entry and storing them as Blobs (or Texts), individually. Another option is to serialise the entire dictionary and store it in a Blob. (I do that, in practice, in the old version of the dictionary application, written in PHP.)
Any advice?
Ikai Lan (Google)
Jan 27, 2012, 6:24:59 PM1/27/12
to google-a...@googlegroups.com
"So, I am considering serialising each dictionary entry and storing them as Blobs (or Texts)"
Going to go out on a limb here ... what about storing the entire dictionary as a single Blob? Or at least the index as a single Blob? You read the index into local instance memory and lazily load the extra data as needed. This makes each lookup take:
1a. (Best case scenario) Local memory index traversal - cost is time in instance hours (very fast), log(8000)
1b. (Worst case scenario) Load the entire index from a single datastore Blob into local instance memory and then traverse (cost of 1a + building in-memory data structure and cost of reading a single entity)
2. Read the necessary entities from the datastore
With only 8000 entries, you should be able to keep most indexes in memory, even within the 128mb limit (I could be wrong). Can you describe some of the searches you need to do in a bit more detail?
--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/KujFnRst_0cJ.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Magnus Gunnarsson
Jan 29, 2012, 3:57:19 AM1/29/12
to google-a...@googlegroups.com
Thanks Ikai.
I'll probably go for your suggestion, and try to cache the indexes in MemCache.
There are different kinds of searches:
1. String search in the spoken language translations of the signs. Most often that is start-of-string, but it could also be end or part of string. There are also three kinds of "translations" that must be kept track of. Sometimes searchers should be restricted to one of these kinds.
2. ID search. Take a sign id, return the sign. I probably won't need a separate index for this.
3. Each sign is transcribed, i.e. there is a structural description of the hand movements. Using this description, similarities between signs can be calculated. So, one search type is to fetch signs that are similar to another sign, in terms of transcription.
MagnusG
I'll probably go for your suggestion, and try to cache the indexes in MemCache.
There are different kinds of searches:
1. String search in the spoken language translations of the signs. Most often that is start-of-string, but it could also be end or part of string. There are also three kinds of "translations" that must be kept track of. Sometimes searchers should be restricted to one of these kinds.
2. ID search. Take a sign id, return the sign. I probably won't need a separate index for this.
3. Each sign is transcribed, i.e. there is a structural description of the hand movements. Using this description, similarities between signs can be calculated. So, one search type is to fetch signs that are similar to another sign, in terms of transcription.
MagnusG
Reply all
Reply to author
Forward
0 new messages