Google Groups

Re: IndexedDB and buddycloud


imaginator Oct 29, 2012 9:57 AM
Posted in group: buddycloud-dev
@parashuram: Excellent! Let's make sure we all understand the problem really well and then your help would really be very welcome.

So I was thinking more about this problem today:
  • read message counters are the crack cocaine of this service. The Symbian client, there was a huge rush and buzz around how many unread stars (we used stars for counters) were on your phone.
  • deal with many posts a popular channel could have a history of 100k messages
  • lots of channels: let's assume that a user follows 500-1000 channels. Some very active, some with no activity for years (olym...@ioc.org)
  • speed to display the first messages to a user
  • failure recover from any network or system outage mid-sync

So let's assume a channel's "truth" on the server is as follows:

TimeStamp PostID
1         abc
2         bba 
3         aaa
4         ddc
5         ada
6         daa
7         cab
8         aca

The client starts with the following messages in it's DB

1         abc
2         bba 
3         aaa

"Send me messages newer than the time "3". The server starts sending 6 messages but due to a network issue only the following 3 messages arrive:

5         ada
8         aca
4         ddc

If we run our "send me messages newer than" now, we'll never receive messages with the timestamp of "6" or "7".

Some questions:
  • am I right to assume then that we have to send messages sequentially to avoid gaps? I was hoping we could avoid a bunch of sorting on the server.
  • @rene: I believe we have the same problem on the Android client: how have you planned on resyncing?
  • @mattW: where does MAM fit into this?
  • what happens when the user has requested the archive for a channel but we also have a new post made (datestamp of "10" in this case)?
  • if we kickstart a fresh install with 50 messages per channel, what happens when we need to scroll backwards (I think MAM should help us here...)
  • how do we sync message revocations to the client?

Lot's of questions... but it's very important that we build something robust (and fast).

If anyone wants to discuss this more we have our daily hangout tonight at 8pm (GMT+1) in https://plus.google.com/hangouts/_/calendar/YnVkZHljbG91ZC5jb21fNDBocmZiMmJ1cmNraXV2ZTdxNG9lOTljb29AZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ.dtg1nbrreu71n9c1h7mbrrdseg

S.


On 29 October 2012 06:47, Parashuram Narasimhan <m...@nparashuram.com> wrote:
> Count me in if you think I could contribute with the IndexedDB work.
>
> Parashuram N.
> http://nparashuram.com
>
>
>
>
> On Sun, Oct 28, 2012 at 8:46 AM, Simon Tennant <si...@buddycloud.com> wrote:
>>
>> Thanks Parashuram. Great input.
>>
>> @buddycloud-guys: what do you think of this as the way to get our message
>> counters working well and an overall snappier client?
>>
>> @sonny: how do you currently store messages in your client
>>
>> Who would like to take the lead on integrating this?
>>
>> In terms of priorities, I suggest we:
>> - finish the launch tasks on trello,
>> - kill any showstopper bugs
>> - add a "give me all my messages in all the channels I follow api call"
>> - add IndexedDB support into the client with the "above API call"
>>
>> Let's talk through this on Monday's G+ Hangout.
>>
>> S.
>> --
>> Simon Tennant | buddycloud.com | +49 17 8545 0880 | office hours
>> goo.gl/tQgxP
>>
>> On 26 Oct 2012, at 22:48, Parashuram Narasimhan <m...@nparashuram.com>
>> wrote:
>>
>> Hi,
>>
>> I think its a great idea to get IndexedDB to cache the content offline - I
>> noticed that you are using Backbone for the project. You could try out the
>> IndexedDB backbone plugin that would make it a lot easier to integrate with
>> minimum code changes. The IndexedDB Backbone adapter is pretty much the same
>> models, except that the models are saved in IndexedDB and can be searched
>> using interesting query parameters. It should require almost no change to
>> your existing models.
>>
>> Also, I was able to get the polyfill working on Phonegap, so you should be
>> able to take your backbone application mobile too..
>>
>> Let me know if you have any questions, I would me more than happy to
>> contribute.
>>
>>
>> Parashuram N.
>> http://nparashuram.com
>>
>>
>>
>> On Thu, Oct 25, 2012 at 4:49 AM, Simon Tennant <si...@buddycloud.com>
>> wrote:
>>>
>>> Parashuram,
>>>
>>> A friend, Sonny (and also a patch submitter of the polyfill shim) turned
>>> me onto indexedDB and onto how we should be storing messages in buddycloud.
>>>
>>> background: budycloud is an open social network (like Twitter, but each
>>> domain can run their own server, and they all interoperate to share updates
>>> from each user).
>>>
>>> We are building the buddycloud webclient
>>> (https://github.com/buddycloud/webclient) and need a way to reliably get and
>>> store posts for each channel ("wall") that a user follows.
>>>
>>> One idea is to use IndexDB for this in buddycloud:
>>> https://groups.google.com/d/topic/buddycloud-dev/ux4uBtpR42E/discussion
>>>
>>> The client will pull down your channel posts, which will mean perhaps
>>> about 20k posts of about the same length as a Google Plus post.
>>>
>>> And finally some questions:
>>>
>>> will this work?
>>> is there anything we should watch for?
>>> can we trigger events on a successful insert or DB change?
>>>
>>>
>>> This looks very promising... just hope we can make it work nicely.
>>>
>>> S.
>>>
>>>
>>>
>>> --
>>>
>>> Simon Tennant | buddycloud.com | +49 17 8545 0880 | office hours:
>>> goo.gl/tQgxP
>>>
>>>
>>
>>
>



--
Simon Tennant | buddycloud.com | +49 17 8545 0880 | office hours: goo.gl/tQgxP