Re: IndexedDB and buddycloud
Simon Tennant
So I was thinking more about this problem today:
- read message counters are the crack cocaine of this service. The Symbian client, there was a huge rush and buzz around how many unread stars (we used stars for counters) were on your phone.
- deal with many posts a popular channel could have a history of 100k messages
- lots of channels: let's assume that a user follows 500-1000 channels. Some very active, some with no activity for years (olym...@ioc.org)
- speed to display the first messages to a user
- failure recover from any network or system outage mid-sync
So let's assume a channel's "truth" on the server is as follows:
TimeStamp PostID
1 abc
2 bba
3 aaa
4 ddc
5 ada
6 daa
7 cab
8 aca
The client starts with the following messages in it's DB
1 abc
2 bba
3 aaa
"Send me messages newer than the time "3". The server starts sending 6 messages but due to a network issue only the following 3 messages arrive:
5 ada
8 aca
4 ddc
Some questions:
- am I right to assume then that we have to send messages sequentially to avoid gaps? I was hoping we could avoid a bunch of sorting on the server.
- @rene: I believe we have the same problem on the Android client: how have you planned on resyncing?
- @mattW: where does MAM fit into this?
- what happens when the user has requested the archive for a channel but we also have a new post made (datestamp of "10" in this case)?
- if we kickstart a fresh install with 50 messages per channel, what happens when we need to scroll backwards (I think MAM should help us here...)
- how do we sync message revocations to the client?
Lot's of questions... but it's very important that we build something robust (and fast).
If anyone wants to discuss this more we have our daily hangout tonight at 8pm (GMT+1) in https://plus.google.com/hangouts/_/calendar/YnVkZHljbG91ZC5jb21fNDBocmZiMmJ1cmNraXV2ZTdxNG9lOTljb29AZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ.dtg1nbrreu71n9c1h7mbrrdseg
S.
On 29 October 2012 06:47, Parashuram Narasimhan <m...@nparashuram.com> wrote:
> Count me in if you think I could contribute with the IndexedDB work.
>
> Parashuram N.
> http://nparashuram.com
>
>
>
>
> On Sun, Oct 28, 2012 at 8:46 AM, Simon Tennant <si...@buddycloud.com> wrote:
>>
>> Thanks Parashuram. Great input.
>>
>> @buddycloud-guys: what do you think of this as the way to get our message
>> counters working well and an overall snappier client?
>>
>> @sonny: how do you currently store messages in your client
>>
>> Who would like to take the lead on integrating this?
>>
>> In terms of priorities, I suggest we:
>> - finish the launch tasks on trello,
>> - kill any showstopper bugs
>> - add a "give me all my messages in all the channels I follow api call"
>> - add IndexedDB support into the client with the "above API call"
>>
>> Let's talk through this on Monday's G+ Hangout.
>>
>> S.
>> --
>> Simon Tennant | buddycloud.com | +49 17 8545 0880 | office hours
>> goo.gl/tQgxP
>>
>> On 26 Oct 2012, at 22:48, Parashuram Narasimhan <m...@nparashuram.com>
>> wrote:
>>
>> Hi,
>>
>> I think its a great idea to get IndexedDB to cache the content offline - I
>> noticed that you are using Backbone for the project. You could try out the
>> IndexedDB backbone plugin that would make it a lot easier to integrate with
>> minimum code changes. The IndexedDB Backbone adapter is pretty much the same
>> models, except that the models are saved in IndexedDB and can be searched
>> using interesting query parameters. It should require almost no change to
>> your existing models.
>>
>> Also, I was able to get the polyfill working on Phonegap, so you should be
>> able to take your backbone application mobile too..
>>
>> Let me know if you have any questions, I would me more than happy to
>> contribute.
>>
>>
>> Parashuram N.
>> http://nparashuram.com
>>
>>
>>
>> On Thu, Oct 25, 2012 at 4:49 AM, Simon Tennant <si...@buddycloud.com>
>> wrote:
>>>
>>> Parashuram,
>>>
>>> A friend, Sonny (and also a patch submitter of the polyfill shim) turned
>>> me onto indexedDB and onto how we should be storing messages in buddycloud.
>>>
>>> background: budycloud is an open social network (like Twitter, but each
>>> domain can run their own server, and they all interoperate to share updates
>>> from each user).
>>>
>>> We are building the buddycloud webclient
>>> (https://github.com/buddycloud/webclient) and need a way to reliably get and
>>> store posts for each channel ("wall") that a user follows.
>>>
>>> One idea is to use IndexDB for this in buddycloud:
>>> https://groups.google.com/d/topic/buddycloud-dev/ux4uBtpR42E/discussion
>>>
>>> The client will pull down your channel posts, which will mean perhaps
>>> about 20k posts of about the same length as a Google Plus post.
>>>
>>> And finally some questions:
>>>
>>> will this work?
>>> is there anything we should watch for?
>>> can we trigger events on a successful insert or DB change?
>>>
>>>
>>> This looks very promising... just hope we can make it work nicely.
>>>
>>> S.
>>>
>>>
>>>
>>> --
>>>
>>> Simon Tennant | buddycloud.com | +49 17 8545 0880 | office hours:
>>> goo.gl/tQgxP
>>>
>>>
>>
>>
>
--
Simon Tennant | buddycloud.com | +49 17 8545 0880 | office hours: goo.gl/tQgxP
Simon Tennant
Hi Parashuram,
So now Abmar has finished the /sync API endpoint: it grabs the newest messages since a timestamp, capped at 50 per channel.
I added a quick wiki page https://buddycloud.org/wiki/Synchronising_posts that shows how we are trying to sync messages into a client indexedDB on the buddycloud client.
The call looks like:
curl --basic -u si...@buddycloud.org:demo "https://api.buddycloud.org/sync?since=2012-09-01T00:00:00Z&max=50"
(basic auth using my test account) and this is the output that we are trying to shove into an indexed DB on the client side:
What do you think the best way to go about adding this to the buddycloud client?
Would you still have some time to help show us or get a framework in place and inserting data to the client ((github.com/buddycloud/webclient)?
I'm really looking forward to meeting too. Let us know if there a more convenient hangout time that doesn't conflict with your scrum call?
Greetings from Germany and Brazil.
S.
Hi,
I would love to join in at 8pm GMT +1, but have a meeting then. Also, I do not have access to the google hangout.
Simon Tennant
Our normal 8pm will be your 11am. We'll be the the regular Google Hangout (https://plus.google.com/hangouts/_/calendar/YnVkZHljbG91ZC5jb21fNDBocmZiMmJ1cmNraXV2ZTdxNG9lOTljb29AZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ.dtg1nbrreu71n9c1h7mbrrdseg) and Rodrigo will be there to talk about the current models.
Shoot me an email if you have any problems joining.
Looking forward to it.
S.
I think a good starting point would be to take a look at the models (https://github.com/buddycloud/webclient/tree/master/js/app/models) and look at the ones we would like to persist in IndexedDB.
Since we have moved to daylight savings, I think we could look at the meeting again. I have daily meetings from 12:00 to 12:30 pm PST, any other time works for me,