Implementing a personalized feed on Google App Engine

[Barada Sahu]

I wonder if anyone has attempted implementing a personalized content feed using GAE. 

The way that I look at it, we will need to pipeline incoming content into buckets and attach them with users based on user's affinity to other content.

The feed generation for the user itself will need to be triggered via events. This may happen pretty often and will lead to a large number of writes. 

Not specifically a use case for which GAE is tuned for (more frequent reads and infrequent writes). 

Wondering if anyone has attempted anything similar or if there are any reference implementation architectures from the Appengine team. 

~ B

[Nick (Cloud Platform Support)]

Hey Barada,

I'm not quite sure exactly what you're looking to implement, and it all sounds good in the abstract. I can speak to some potential concrete issues, though:

1. The advice to tune the frequency of reads to be higher than writes is specific to the Datastore, not App Engine in general. You could easily use another storage solution of all different kinds (files on Compute Engine disks, Cloud Storage, Cloud SQL, Memcache, BigTable, etc.)

2. Even with Datastore as the storage engine, you could easily write many entities at a time without problem - the restrictions on writes / updates are on entity groups. You could then query the entities for matching tags as a repeated string property (python docs, but other languages have similar docs), returning the matching tagged results to the user, having created the query based on their affinities.

I hope this has been helpful in thinking about designing your system,

Nick
Cloud Platform Community Support

[Kaan Soral]

"personalized content" is complex regardless of GAE

I have chronological + geo feeds, for each event, an entity is written for each person

Then, that person fetches those entities while digesting the feed

IMO, whatever you do, you have to pre-process things with GAE, as the on-request capacity is pretty low, it's not logical to dynamically generate a feed in a timely manner 

[Nick (Cloud Platform Support)]

Yep, Kaan's idea is very good. Pre-calculating the results and periodically updating them is a great method, but knowing how to avoid stale results, that is, knowing when to run the next dynamic query, is a fine art to make sure users' experience is "live". 

[Barada Sahu]

Thanks Kaan, Nick,

Eventually the solution I have implemented is akin to Kaan's in the sense that there is event driven feed generation for users. 

However since there are a bunch of users I cannot use a tag based solution as I would have to keep writing pretty often to the same table in Datastore (assuming a tag specific to each user). Instead I make a copy of the data element and attach it to the user's feed table. This makes the feed table grow pretty large as I have a bunch of copies of the same item across multiple feed models for users. It's a tradeoff between writing to the content table (frequently) vs user-feed table (less frequently). 

With the NDB library am using to access Datastore so it's already cached. The feed generation task although times out quite often on the AppEngine. I use auto-scaling and suspect this is the one causing issues, should possibly go onto a more managed service environment. 

More importantly I wanted to have a discussion on using Datastore as a storage engine and the data-model for writing a personalized feed and possible read-write impact of storing the feeds on the datastore. As it is a personalized feed am assuming there is a feed (model) per each user. How do I structure the associations of content-user in the feed model?

~ B