Firestore pagination with snapshot listeners

[basti skalendudler]

I am working with Firestore right now and have a little bit of a problem with pagination.
Basically, I have a collection (assume 10 items) where each item has some data and a timestamp.

Now, I am fetching the first 3 items like this:

Firestore.firestore()
.collection("collectionPath")
.order(by: "timestamp", descending: true)
.limit(to: 3)
.addSnapshotListener(snapshotListener())
Inside my snapshot listener, I save the last document from the snapshot, in order to use that as a starting point for my next page.

So, at some time I will request the next page of items like this:

Firestore.firestore()
.collection("collectionPath")
.order(by: "timestamp", descending: true)
.start(afterDocument: lastDocument)
.limit(to: 3)
.addSnapshotListener(snapshotListener2()) // Note that this is a new snapshot listener, I don't know how I could reuse the first one
Now I have the items from index 0 to index 5 (in total 6) in my frontend. Neat!

If the document at index 4 now updates its timestamp to the newest timestamp of the whole collection, things start to go down.
Remember that the timestamp determines its position on account of the order clause!

What I expected to happen was, that after the changes are applied, I still show 6 items (and still ordered by their timestamps)

What happened was, that after the changes are applied, I have only 5 items remaining, since the item that got pushed out of the first snapshot is not added to the second snapshot automatically.

Am I missing something about Pagination with Firestore?

[Kato Richardson]

Hello Basti,

Can you provide a more complete code sample and some sample data, maybe in a gist or such? It's kind of hard to follow exactly how and when your data is getting manipulated.

☼, Kato

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.
To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/ca247330-5472-4742-81e3-06cbbe3829a4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Kato Richardson |

Developer Programs Eng |

kato...@google.com |

775-235-8398

[Kato Richardson]

Hello Basti,

I had to go learn how this works to answer your question. Sorry for the delay : ).

I think the client is behaving correctly here. The problem is the "startAfter" clause in your second query. Let's assume documents ordered descending by timestamp, with ids F, E, D, C, B, A:

So in your example, you have F,E,D in your first query. And then you start your second with startAfter=D, so the inflection point is indicating in blue here:  F, E, D, C, B, A

If you then give C the newest timestamp, you get a list like the following:  C, F, E, D, B, A. But notice that the inflection point is still startAfter=D. So it makes sense that the second list now only contains B, A.

Does that help at all?

☼, Kato

On Thu, Nov 9, 2017 at 12:20 AM, basti skalendudler <skalde...@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.
To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/ca247330-5472-4742-81e3-06cbbe3829a4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[basti skalendudler]

hey Kato, I am not sure whether my last message has been sent, google groups is not very intuitive to use imho ^^

Thank you very much for investigating my problem!

I understand the reason why it works the way it works, but is this now the end of it? I mean, I thought pagination + listening for changes 

in all pages was a standard use case... Do you have an idea how I could make this work?

Basti

To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/ca247330-5472-4742-81e3-06cbbe3829a4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Samuel Stern]

Hi Basti,

Now that we've determined the SDK is behaving properly, what's left is a documentation/library issue.  As you can see implementing infinite scroll with real-time data is really tricky because the data points tend to jump between "pages".  There's no magic bullet, just really careful code.  I plan to add a page like this to our "solutions" section of the docs and possibly implement a reference implementation in FirebaseUI.

Once we've done that I will share on this mailing list.  And of course if you or anyone else comes up with a solution that you like, please share!

- Sam

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/d066c7e9-dc44-410c-9ca4-e9db9fbac900%40googlegroups.com.

[basti skalendudler]

Hey Sam,

You are correct, the SDK behaves correct, though, this feature would be really nice.

I stumbled over all of this because we are implementing a simple chat right now for our app.

And the first idea that came to my mind was:

"Sure, if somebody has a lot of chats, you must use pagination to not retrieve all of them at once. But what to do if an old chat gets revived by a new message?

Yeah, of course, just apply a listener to each page and handle the incoming change events"

And it almost worked, just this last insert in the second page (see the example above) is the missing thing here.

Now, you said one could do a very careful implementation to get this feature to work. The only idea I have right now, would be to reload every subsequent page

of the page that receives the new element.

In our example, that would mean: "Hey, I inserted an element in my page and removed the last one. That means, that each subsequent page now must be reloaded 

to reflect these changes" 

Do you have a more sophisticated approach? Because this would mean a huge network data impact, if you have big/many pages to reload every time.

- Basti

[thereb...@gmail.com]

I'm also having same issue i want pagination with snapshot listener so that i can satisfy my user from both sides pagination and new upcoming message. If you found any solution then kindly tell us.

[Milan Agarwal]

Hey guys,

I came across the same use case today and I have successfully implemented a working solution in Objective C client. Below is the algorithm if anyone wants to apply in their program and I will really appreciate if Firestore team can put my solution on their page.

Use Case: A feature to allow paginating a long list of recent chats along with the option to attach real time listeners to update the list to have chat with most recent message on top.

Solution: This can be made possible by using pagination logic like we do for other long lists and attaching real time listener with limit set to 1:
Step 1: On page load fetch the chats using pagination query as below:

- (void)viewDidLoad {
    [super viewDidLoad];
    // Do any additional setup after loading the view.
     [self fetchChats];
}

-(void)fetchChats {
    __weak typeof(self) weakSelf = self;
     FIRQuery *paginateChatsQuery = [[[self.db collectionWithPath:MAGConstCollectionNameChats]queryOrderedByField:MAGConstFieldNameTimestamp descending:YES]queryLimitedTo:MAGConstPageLimit];
    if(self.arrChats.count > 0){
        FIRDocumentSnapshot *lastChatDocument = self.arrChats.lastObject;
        paginateChatsQuery = [paginateChatsQuery queryStartingAfterDocument:lastChatDocument];
    }
    [paginateChatsQuery getDocumentsWithCompletion:^(FIRQuerySnapshot * _Nullable snapshot, NSError * _Nullable error) {
        if (snapshot == nil) {
            NSLog(@"Error fetching documents: %@", error);
            return;
        }
        ///2. Observe chat updates if not attached
        if(weakSelf.chatObserverState == ChatObserverStateNotAttached) {
            weakSelf.chatObserverState = ChatObserverStateAttaching;
            [weakSelf observeChats];
        }
       
        if(snapshot.documents.count < MAGConstPageLimit) {
            weakSelf.noMoreData = YES;
        }
        else {
            weakSelf.noMoreData = NO;
        }
       
        [weakSelf.arrChats addObjectsFromArray:snapshot.documents];
        [weakSelf.tblVuChatsList reloadData];
    }];
}

Step 2: On success callback of "fetchAlerts" method attach the observer for real time updates only once with limit set to 1.

-(void)observeChats {
    __weak typeof(self) weakSelf = self;
    self.chatsListener = [[[[self.db collectionWithPath:MAGConstCollectionNameChats]queryOrderedByField:MAGConstFieldNameTimestamp descending:YES]queryLimitedTo:1]addSnapshotListener:^(FIRQuerySnapshot * _Nullable snapshot, NSError * _Nullable error) {
        if (snapshot == nil) {
            NSLog(@"Error fetching documents: %@", error);
            return;
        }
        if(weakSelf.chatObserverState == ChatObserverStateAttaching) {
            weakSelf.chatObserverState = ChatObserverStateAttached;
        }
       
        for (FIRDocumentChange *diff in snapshot.documentChanges) {
            if (diff.type == FIRDocumentChangeTypeAdded) {
                ///New chat added
                NSLog(@"Added chat: %@", diff.document.data);
                FIRDocumentSnapshot *chatDoc = diff.document;
                [weakSelf handleChatUpdates:chatDoc];
               
            }
            else if (diff.type == FIRDocumentChangeTypeModified) {
                NSLog(@"Modified chat: %@", diff.document.data);
                FIRDocumentSnapshot *chatDoc = diff.document;
                [weakSelf handleChatUpdates:chatDoc];
            }
            else if (diff.type == FIRDocumentChangeTypeRemoved) {
                NSLog(@"Removed chat: %@", diff.document.data);
            }
        }
    }];
   
}


Step 3. On listener callback check for document changes and handle only FIRDocumentChangeTypeAdded and FIRDocumentChangeTypeModified events and ignore the FIRDocumentChangeTypeRemoved event. We are doing this by calling "handleChatUpdates" method for both FIRDocumentChangeTypeAdded and FIRDocumentChangeTypeModified event in which we are first trying to find the matching chat document from local list and if it exist we are removing it from the list and then we are adding the new document received from listener callback and adding it to the beginning of the list.


-(void)handleChatUpdates:(FIRDocumentSnapshot *)chatDoc {
    NSInteger chatIndex = [self getIndexOfMatchingChatDoc:chatDoc];
    if(chatIndex != NSNotFound) {
        ///Remove this object
        [self.arrChats removeObjectAtIndex:chatIndex];
    }
    ///Insert this chat object at the beginning of the array
     [self.arrChats insertObject:chatDoc atIndex:0];
   
    ///Refresh the tableview
    [self.tblVuChatsList reloadData];
}

-(NSInteger)getIndexOfMatchingChatDoc:(FIRDocumentSnapshot *)chatDoc {
    NSInteger chatIndex = 0;
    for (FIRDocumentSnapshot *chatDocument in self.arrChats) {
        if([chatDocument.documentID isEqualToString:chatDoc.documentID]) {
            return chatIndex;
        }
        chatIndex++;
    }
    return NSNotFound;
}


Step 4. Reload the tableview to see the changes.

This is a working solution in my project and anyone having issues implementing it or understanding it can let me know. I will be pleased to help them out :)

Thanks,
Milan Agarwal

[Will Battel]

Thanks Milan, I'm going to try this out!

I'm working on a project in Swift with the same Firestore pagination requirements as this thread describes. I almost got it working perfectly, but hit what seems to be the same issue as the original author of this thread. (See here: https://stackoverflow.com/questions/54274703/populating-uitableview-from-firestore-with-query-cursors-paging)

I'm going to try this out, although my Obj-C is a little rusty. I hope the Firebase team can take this and add some meaningful documentation for this common use-case.

I'll add an update with my results after attempting to implement this.

[Will Battel]

Milan, after closer inspection, I've found that your solution does not quite solve the problem for the original poster (OP) of this thread or my own scenario.

In your chat app you assume (appropriately) that documents will not change after being created. However, both the OP and I have scenarios that cannot make that assumption. My app, for example, is displaying "posts" that can have their content changed after creation. This means I have to listen to snapshot changes on each "page" that I query. Your solution depends on the getDocuments fetch that will not allow for the near-real-time updates I am seeking.

It's still a nice solution for appropriate use-cases, and I thank you for sharing, but unfortunately it does not work for the OP or I because we need the snapshot listener for all of the documents, not just the most recent one.

Again, just for reference, here is where I discuss my challenge: https://stackoverflow.com/questions/54274703/populating-uitableview-from-firestore-with-query-cursors-paging

I'm hoping someone from the Firebase team can pitch in here and help solve this problem for what I'd imagine will be a common use case with Firestore. Not even FirebaseUI has paging support yet.

[Milan Agarwal]

Hello Will, sorry for replying late here as I didn't get any notification for this. I my use case I also considered that a document can change and it is changing once it's created but I am leveraging timestamp to handle this.

The main thing is whenever my document get's changed I update the timestamp as well on which my listener depend and thus for every document change I receive the listener callback.

I read your stack overflow question as well and in your use case you can do the same what I am doing above i.e Use one more property named as updatedAt and attach the listener with limit set to 1 on this property. Now you have to ensure that whenever a change in document is made you have to update this timestamp as well and thus when this timestamp gets updated you listener callback will be called.

Let's take an example, suppose you have 5 posts already P1, P2, P3, P4, P5 so they will be displayed in your UI as

P5

P4

P3

P2

P1

Now if one more document P6 gets created and you set it's createdAt and updatedAt timestamp on creation as well, your listener will be called and using my logic mentioned above will attach the document to the top of the list. The updated list will be:
P6

P5

P4
P3
P2

P1
Now suppose documents P3 and P2 gets updated but P3 is updated slightly after P2 so now the callback will be called twice once for P2 and then for P3 which will finally make the list look like:
P3
P2
P6
P5
P4
P1

I hope it helps you to understand now.

Thanks,
Milan

[Dmitriy Kulakoff]

Hi Milan, thx for your solution, but I have some questions: 

- What you will do if messages will updated at the same time?

- How your handle updateAt field if message was deleted?

[Milan Agarwal]

Hi Dmitriy,

You have to extend my solution if you also want to handle message update and delete, just the same way I explained to Will in my previous message of the same thread. You only have to handle this for the currently open chat, so all you can do is have one more field named as updatedAt on message collection and attach one more listener for updatedAt, this way if a message is updated it's updatedAt field would have been updated and your listener will get called using which you can update your local list.

Delete is also a special case of update where you have update the message document with an identifier saying message is deleted or you can simply update the message text saying this message is removed.

I hope this may help you :)

Thanks,

Milan Agarwal