Best datastore design for massively scalable realtime chat?

[Joonas Pihlajamaa]

Hi all,

I'm trying to build chat functionality into my Google App Engine web application. The basic idea is that User loads a chat page that employs AJAX calls to poll new chat messages in the channels he/she is following every few seconds, and user can also write to the channels using AJAX.

1. A simple channel entity with a list property containing message strings is OK for channels with a few users and infrequent updates, but once a channel becomes "hot" (likely to happen for the purpose I have in mind), there's suddenly 300 users polling a single channel entity every few seconds, and new lines being added every few seconds. Once the entity contains something like 1000 message strings, this starts to sound a bad idea.
2. Creating a new entity type for message line with timestamp is also possible, but polls would then be of type "give me all messages since timestamp X", and every poll would need a filtered query into message pool, and with a thousand users this would mean something in the ballpark of 200 queries per second. Also, this solution doesn't play well with memcache. Changing timestamp to incremental counter would improve the situation, but I still don't know if it's efficient to cache each chat message as a single object in memcache?
3. The solution I'm now considering would be to "page" a chat channel, so one chat page entity would contain something like 50 messages at most (as a list type property), and when that limit is reached, a new page is created. Each channel object would have a reference to the current page and previous page, so new users joining the channel would have quick access to >50 of the latest messages. Pages would also lend themselves well for memcaching. A poll would then be of type "If have the messages until page 3, message 23 - what's new?".
   

Any ideas or comments on the different options? Am I worrying too much about this? I know this currently is not issue, but my aim is that the data structure should gracefully and efficiently handle something like 2500 simultaneous users, each polling typically one to ten different channels and contributing to one or two.

[Barry Hunter]

AppEngine actully has XMPP intergtation
http://code.google.com/appengine/docs/python/xmpp/

And the Channel API
http://code.google.com/appengine/docs/python/channel/

which should eliminate the need for polling AJAX requests yourself.

Could probably use the matcher API (renamed Speculative Search API)
http://groups.google.com/group/google-appengine/browse_thread/thread/5462e14c31f44bef
to perform dispatching of messages, again eliminating the need for the
polling on the datastore.

On 23 February 2011 08:29, Joonas Pihlajamaa

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

[Joonas Pihlajamaa]

Thank you for the _very_ useful and informative reply. :)

> AppEngine actully has XMPP intergtationhttp://code.google.com/appengine/docs/python/xmpp/

Yes, but from I can gather, I would need to run an XMPP server
somewhere to handle the chats (or find a free provider X), and because
chat is not the main function of my app (I'm building a go board game
server) and needs to be integrated with the rest of the application,
it's not the ideal choice.

> And the Channel API http://code.google.com/appengine/docs/python/channel/

This was actually extremely interesting and I think I'll consider this
to get rid of the polling. However, because each user can only have
one channel open, I need to do similar logic to polling on server-side
anyway (i.e. merge messages from different channels), so my data
structure problem still remains.

> Could probably use the matcher API (renamed Speculative Search API)http://groups.google.com/group/google-appengine/browse_thread/thread/...

> to perform dispatching of messages, again eliminating the need for the polling on the datastore.

It seems this is still in very early testing phase. Because I don't
yet have the user numbers that would require this infrastructure, I
think I'll be doing without for the time being, and rewrite the
merging functionality using matcher API when it's open to all App
Engine users. (Don't think I should apply for testing when I only have
a few concurrent users at most :)

[Barry Hunter]

On 23 February 2011 14:20, Joonas Pihlajamaa

<joonas.p...@gmail.com> wrote:
> Thank you for the _very_ useful and informative reply. :)
>
>> AppEngine actully has XMPP intergtationhttp://code.google.com/appengine/docs/python/xmpp/
>
> Yes, but from I can gather, I would need to run an XMPP server
> somewhere to handle the chats (or find a free provider X),

Google Talk :)

http://www.google.com/support/talk/bin/answer.py?hl=en&answer=57677

(I beleive XMPP and the channel API already use the Talk infestructure
under the hood anyway)

> (Don't think I should apply for testing when I only have a few concurrent users at most :)

Testing in a few users, is still worthwhile. Get it working on a few,
and it should work for more.

You might not even need to apply now. Speculative Search should be
near ready for launch (it didnt make the latest SDK release tho)

[Robert Kluin]

Speculative Search + channel API might be a really nice / cool
solution. It would more-or-less handle a lot of the routing for you.

If you want users to be able to view / page through old 'chat logs'
then, as you mentioned, you'll need to store them. How fast do you
expect to receive messages for a given channel, how large are the
messages expected to be on average, and how much history do you want
to keep for users to view? If messages will typically arrive less
than one per 1/2 second or so, option (1) or (3) might work really
nicely. If messages are small and you only need, say, the last 100
messages your first option might work very well.

Keep in mind, if you will not be querying the list property, you can
serialize and store you data significantly more efficiently in a text
or blob property.

Obviously storing each message in its own entity makes writes simple,
but means you need a query to send messages to users instead of a
fetch by key. If your write rate to a channel is low enough, I
suspect option (1) or (3) will make reads extremely fast. If the
write rate is high the second option might be your best choice, or
some combination approach (ie write new messages to their own entities
and run a background task to batch up new messages and update the
'channel' entities).

Just a few thoughts.

Robert