Getting posts

[Simon Tennant]

Hi all,

We were talking in the daily hangout about syncing and unread counters for the new webclient.

A couple of thoughts in no particular order:

• speed. Fast initial page load with data. Most important is to quickly paint the channels followed list so that the user can start clicking on a channel.
  
• We need to add a new api call that retrieves all posts since a timestamp (I think we have this on the XMPP side but not on the HTTP API)
• the idea of having an huge unread count per channel doesn't make sense. For example if you followed me on Facebook, you would see an unread counter of 8000. But that really doesn't make sense - sure there are 8000 posts there but are you really going to scroll back to 1997 to read all of them.
  I think we need to rethink our unread count and have two states. 0-50 unread posts and "lots" per channel. That saves the user from needing to scroll through a huge bunch of messages to clear their unread counter, but also gives them enough context in a channel when they join.
• work towards storing messages between session on the client. We currently do this on the Android (1) and Symbian (2) clients.
• use indexedDB (http://caniuse.com/#search=indexeddb) for client side storage with the Polyfill Shim for Safari+websql). (http://blog.nparashuram.com/2012/05/introducing-indexeddb-shim-over-websql.html). This gives us about 50MB of storage which should be more than enough for a while (famous last words etc etc)

Client flow: first login

1. Pull subscribed channels and store that in indexedDB
2. we get a rough amount of messages to semi populate the client. (~ GET https://api.example.com/all-posts-since<now>&limit=(50 x <number of channels followed>)  - off the top of my head.... 
3. if you click on a channel and there are less than 50 posts, more posts are pulled in for that channel (via the existing infinite scroll function)

Client flow: subsequent login.

1. look in indexedDB and find the most new post and asks for any posts newer. (what happens if the user has been offline for 6 months?)
2. newer posts are delivered via the API and the counter is increased continuously
   

Calculating an unread count per channel

1. Lets say a user follows an existing channel foo@domain, our unread count would be 50
2. 10 messages are displayed on the screen, our unread counter drops to 40
3. they browse to another channel.
4. a new message is posted to foo@domain, the counter is incremented and now stands at 41.

How does this sound?

S.

1. https://github.com/buddycloud/android-client/blob/master/src/com/buddycloud/content/BuddycloudProvider.java#L66
2. https://github.com/buddycloud/symbian-client/blob/master/src/BuddycloudLogic.cpp (don't ask me where... )



--

Simon Tennant | buddycloud.com | +49 17 8545 0880 | office hours: goo.gl/tQgxP

[Rodrigo Duarte]

I just have one doubt, what happens if I have 1000 unread messages in a particular channel? So we would show something like 50+? And how we would decrement the counter in this case?

--
 
 

--
Rodrigo Duarte Sousa
MSc candidate in Computer Science

Distributed Systems Laboratory
Federal University of Campina Grande
Campina Grande, PB - Brazil
http://lsd.ufcg.edu.br/~rodrigods

[Simon Tennant]

@rodrigo: good idea about the 50+ display

Perhaps we could mark the 950 messages as "read" in our client-side DB. Then display "50+" on the client. When any of the 50 unread messages are displayed on screen we mark them as "read"

IMHO, I think the best approach will be to have a DB locally that we can fill up and then experiment with different display options. 

We have two states that we need to deal with:

1. never logged into bc on this web client before

- create a new indexedDB on the client

- get the 51 most recent messages for each channel they follow

- all channels with >50 unread messages get the "50+" unread counter.

- infinite scroll and pull in messages from history

2. has logged in before and there is some history (for that user account) in the indexedDB.

(this one is more tricky: if sync-ing fails half way through we will never know what timestamp to resync from again)

aim: show the user's home channel first with posts

aim: pull in other channels posts in the background 

- display example.com/m...@example.com  with their existing posts

- request messages newer than most recent

- store these messages with an extra flag "sync"

- when we have a complete archive for that channel remove the "sync" flag from the new messages (is this the best way to protect against a failure mid-sync?)

- update the user's personal channel with the new posts

- increment the counter on m...@example.com

- … somehow get other posts but not the user's personal channel posts?

So thinking through this, perhaps we need an API call like "give me all new posts since $timestamp AND first send me my personal channel's posts"

This all seems very clunky… Perhaps someone else have a cleaner more elegant solution to quickly display new data to us when we log in.

S.

-- 

Simon Tennant | buddycloud.com | +49 17 8545 0880 | office hours goo.gl/tQgxP