Publish-subscribe to dynamic recipient lists
PEZ
thinking. It seemed to work at first, but then, since the IN operator
works the way it does, we quickly realized we couldn't continue on
that path. We found Brett Slatkin's presentation from last years
Google I/O and we have now watched it a few times and also played some
with the http://pubsub-test.appspot.com/ example but it isn't clear to
us how to do it with "dynamic" recipient lists.
We want to implement a "News feed" where a user can see messages
broadcasted by her friends, sorted with newest message first. But the
feed should reflect changes in her friends list. (If she adds new
friends, messages from those should be included in the feed, and if
she removes friends their messages should not be included.) If we use
the pubsub-test example and attach a recipient list to each message
this means a lot of manipulation of the recipient lists when users
connect and disconnect friends.
What we need are some hints on how to "think" when modeling this. Does
this description even make sense?
Ikai L (Google)
- is removing of friends a common event? similarly, is adding of
friends a common event? (All relative, relative to "reads" of the news
feed)
- From what I remember, the only way to make heavy reads scale is to
write the data multiple times in peoples' streams. Twitter does this,
from what I remember, using a "eventually consistent" model. This is
why your feed will not update for several minutes when they are under
heavy load. The general consensus, though, is that a relational,
normalized model simply will not work.
- the Jaiku engine is open source for your study:
http://code.google.com/p/jaikuengine. This runs on App Engine
Hope these help when you're considering a design.
> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
>
>
--
Ikai Lan
Developer Programs Engineer, Google App Engine
http://googleappengine.blogspot.com | http://twitter.com/app_engine
PEZ
friends lists is uncommon. We've sort of settled for an "eventually
consistent" model now where we update the recipient lists of the
freshest messages immediately and queue the update of older messages
using the Tasks API. Really good to know that this path is tried and
proven before us. Thanks! And thanks for the jaikuengine link.
Strange, I actually knew it was open sourced, but didn't think about
exploring it. Yes, you really helped us in our design considerations!
/PEZ
On Mar 16, 8:00 pm, "Ikai L (Google)" <ika...@google.com> wrote:
> A couple of thoughts here:
>
> - is removing of friends a common event? similarly, is adding of
> friends a common event? (All relative, relative to "reads" of the news
> feed)
> - From what I remember, the only way to make heavy reads scale is to
> write the data multiple times in peoples' streams. Twitter does this,
> from what I remember, using a "eventually consistent" model. This is
> why your feed will not update for several minutes when they are under
> heavy load. The general consensus, though, is that a relational,
> normalized model simply will not work.
> - the Jaiku engine is open source for your study:http://code.google.com/p/jaikuengine. This runs on App Engine
>
> Hope these help when you're considering a design.
>
>
>
>
>
> On Mon, Mar 15, 2010 at 5:32 AM, PEZ <p...@pezius.com> wrote:
> > We first modeled publish-subscribe "fan out" using conventional RDBMS
> > thinking. It seemed to work at first, but then, since the IN operator
> > works the way it does, we quickly realized we couldn't continue on
> > that path. We found Brett Slatkin's presentation from last years
> > Google I/O and we have now watched it a few times and also played some
> > with thehttp://pubsub-test.appspot.com/example but it isn't clear to
> > us how to do it with "dynamic" recipient lists.
>
> > We want to implement a "News feed" where a user can see messages
> > broadcasted by her friends, sorted with newest message first. But the
> > feed should reflect changes in her friends list. (If she adds new
> > friends, messages from those should be included in the feed, and if
> > she removes friends their messages should not be included.) If we use
> > the pubsub-test example and attach a recipient list to each message
> > this means a lot of manipulation of the recipient lists when users
> > connect and disconnect friends.
>
> > What we need are some hints on how to "think" when modeling this. Does
> > this description even make sense?
>
> > --
> > You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> > To post to this group, send email to google-a...@googlegroups.com.
> > To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
> > For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en.