Store list in google datastore

[vinvar k]

Hi,

I would like to store a ArrayList<Object> in datastore. The values that are supported to be stored in google datastore does not contain ArrayList .But still im able to save it by just calling 

entity.put().

Is there any limit for the amount of data that can be as a ArrayList in datastore or no limit and i can store a list of 1000's object wihout any worry.

[Vinny P]

On Mon, Jul 14, 2014 at 7:40 AM, vinvar k <vinv...@gmail.com> wrote:

I would like to store a ArrayList<Object> in datastore. The values that are supported to be stored in google datastore does not contain ArrayList .But still im able to save it by just calling entity.put().

You can save a Collection (or implementations of Collection) to the datastore: https://developers.google.com/appengine/docs/java/datastore/jdo/dataclasses#Collections . Fortunately for your use case, ArrayList implements Collection. 

On Mon, Jul 14, 2014 at 7:40 AM, vinvar k <vinv...@gmail.com> wrote:

Is there any limit for the amount of data that can be as a ArrayList in datastore or no limit and i can store a list of 1000's object wihout any worry.

Well, there is the usual limit of 1 MB for a datastore entity - if you're storing thousands of entries, you'll most likely hit that limit sooner or later. But more to the point, you may want to rethink your schema if you're intending to save thousands of values into a property. Usually (but not always) there's a simpler way to store what you need.

 

 

-----------------

-Vinny P

Technology & Media Consultant

Chicago, IL

App Engine Code Samples: http://www.learntogoogleit.com

[Grant Jason]

Your responses are always so useful Vinny, thanks. 

Can you please give us an example or two, wrt re-designing your schema to avoid running into the 1MB limit for a List property. 

Would love to know you recommend here that works with the Datastore.

[Vinny P]

On Thu, Jul 17, 2014 at 4:36 AM, Grant Jason <grant...@gmail.com> wrote:

Can you please give us an example or two, wrt re-designing your schema to avoid running into the 1MB limit for a List property. 

Would love to know you recommend here that works with the Datastore.

Well, the problem is that the right schema is so heavily dependent on what type of app is being written, and what the data being stored looks like. But here's a quick example:

Some applications need only a short list of values in a single property name. For example, a simple blogging app could use an Entity similar to the below for storing blog posts:

Kind: blogpost

ID: [number allocated by the datastore]

Properties:

blogtitle: "Hi!"

blogtext: "This is my first post on my new blog."

creationdate: 1405643123460 (number of milliseconds since UNIX epoch)

tags: "blog" , "google" , "myfirstpost"

Obviously, the list of tags on a blog post are going to be very short; realistically, you'll never have thousands of tags on a single blog post. So in cases such as this using a list property is fine.

The problem comes when a list could be very, very long. Consider an example app designed to hold a list of the user's favorite bookmarks. A problematic entity could look like this:

Kind: Bookmarks

ID: [number allocated by the datastore]

Properties:

email: "vi...@example.com"

favoritelinks: " http://movies.disney.com/frozen " , " learntogoogleit.com " , " https://www.google.com/finance "

The problem is that some people have a lot of bookmarks, and URLs can get very long. It's very, very easy to have over 1 MB of bookmark links. Instead of that, create an entity per bookmark, then map the bookmark as the child of the user entity. For instance, you could do this:

Kind: UserBookmarks

ID: 1 [number allocated by the datastore]

Properties:

email: "vi...@example.com"

(1 bookmark entity per link, you can set ancestors like this: Entity bookmark = new Entity("Bookmark", id, user_entity.getKey()); )

Kind: Bookmark

Ancestor: User:1 (above entity)

ID: [number allocated by the datastore]

bookmarklink: " http://movies.disney.com/frozen "

bookmarktitle: " Frozen "

bookmarkdate: 1405643123460

From there, it's easy to query by setting the ancestor (the user vi...@example.com). This is also a superior arrangement than before since now metadata can be added about the link.

There are other ways to handle this as well. Some developers don't like using ancestor/child relationships; in that case you could skip setting the ancestor on the Bookmark entity, but add a property listing the user the bookmark belongs to: user: vi...@example.com (or the ID of the user). Then query by adding an AND user=useremail/ID property to the query.

'

Another option is to use namespacing. Declare a namespace per user ( https://developers.google.com/appengine/docs/java/multitenancy/multitenancy#Java_Creating_namespaces_on_a_per_user_basis ), then add an entity per bookmark. But the entities don't need to list an ancestor or an owning user because the namespacing service will automatically know which namespace (the username of the user) the entity belongs to.

Some lists can be dramatically shortened by using IDs instead of exact names. For example, suppose an entity stored a list of books by name. That list of names could be replaced by instead listing the ISBN number of the books. This makes the list much shorter and simpler to debug.

With that said, a caveat: in cases such as this, it's always important to reevaluate your storage options. Some people try to force their data to fit into the datastore when purchasing a Cloud SQL instance would be better. Always consider your data and the queries you run: some queries and schema will simply be cheaper with Cloud SQL even though the datastore has a free quota. There is nothing wrong with an application using both the datastore and Cloud SQL services: complex applications often do that.

--------------------------------

Last item: this email turned out longer than I wanted it to be, and I glossed over a lot of important topics and discussion (bolded for a reason!). If you're interested in this topic, I suggest getting a copy of Programming App Engine: http://www.amazon.com/Programming-Google-App-Engine-Sanderson/dp/144939826X . It covers a lot about the datastore and I believe the author is a Google engineer himself, so you're getting the facts straight from the proverbial mouth :-)

I would also suggest 7 Databases in 7 Weeks (covers NoSQL, not the App Engine Datastore directly but it's useful) and Database Design For Mere Mortals (covers relational design, still useful).

And finally (whew, this is getting out of hand!) to any college students and fresh graduates reading this: database design and management is a hard, but very fun subject. It's also a well paying job, if that's something that interests you.....