Firestore Data Structure for Chat App

[Minh Duc Trinh]

Currently I am a one-man team working on a Chat App using Flutter as Front-end and FireBase as backend. I store the data using Firestore. As of now I have 3 collections: Users, Groups and Messages:

• Users: this collection is for storing user data. Each document represent an user with field such as userId, name, image,... Each document also have a subcollection Contacts that store the partial user data of users in the current document Contacts, each sub-document contain info like userId, nickname, image,...
• Groups: this collection is for the a Group Chat, which divided into 2 types using Type field in each document: one-to-one chat and group chat. A document also contains info such as groupId, groupName, ... as well as an array of userId that the group has as Members
• Messages: each document in this collection contains only one subcollection and named after the Group in which the Messages document belong to. In the subcollection is the individual message of a group, which has the field: sentBy is the userId value, sentAt is the timestamp, the messageContent and the type of message

Everything is working as intended as of now, but I stumble upon a problem: When an user enter a Group Chat, the app get the information of each user using the Members array of the group and, along side with the Messages collection, display each user's messages in the chat. But when an user left the group or was removed, so as their userId in the Members array and the app cannot display that user info. I was thinking of getting the user info of each message base on the userId stored in the sentBy field but if that user chat a lot then it will perform a lot of read to Firestore which is not desirable. The other 2 ways I have thought about is adding a group array field to each user document to display the groups that the user is in, and in the group document change the Members array into a Map or Array of Maps with each element in the Map or Array containing the userId as well as the status of the user (remove or active) and maybe a field for the user's role in the group but I don't know if this is optimal.

Can you guys suggest me a way to best suit my current problems?

[Kato Richardson]

Hi Minh,

There are a few different approaches you can take, which are essentially tradeoffs between massive scale and complexity of code.

1. Simple (for chats with members measured in hundreds): Keep the user metadata in some sort of users collection. Look up chat participants as needed when you need their info, but memoize the list (cache it locally) so they are only fetched once.
2. Hard but scalable (for chat apps with members in the thousands): Denormalize by putting user metadata into the messages or groups. This is the approach apps like Gmail take. If you've ever had a friend change their name in gmail and gone to an old message in your archive from them, you can see that the name is still the old one attached to that record and never updates. You can of course find ways to update the metadata if the master copy changes as well, but you're again increasing complexity and reducing scale.

Also check out Todd's excellent series on Getting to Know Firestore, which covers all the nuances of data structures and database architecture.

☼, Kato

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/3b4c1b63-c119-4b9e-b663-ff32d49ecd7dn%40googlegroups.com.

--

Kato Richardson |

Developer Programs Eng |

kato...@google.com |

775-235-8398

[Minh Duc Trinh]

Hey Kato,

Thanks for your suggestion, however I see some problems with those 2 approach:

• Caching the user info locally: this work if the user is in the group chat before another leave, then one person leave and the info still cache in the device thus the app is able to get. However, for users that join the group after that user has left or removed, then they cannot get the user info since that user's id is no longer in the Members array of the group and they have no info of that user cached in their device.
• Storing the metadata of user's message seperately: I also considered this, however if not implement a way of mass update when the master data change, then if the user change their name or avatar image multiple time while chatting, then it would appear in the chat to other user as multiple different users chatting.

I have already watched Todd's video on Data Structure in Firestore and working on restructuring my database but have yet found an optimal solution. My current one is to store all the userId that has been added to the group in a Map as key and differentiate their roles (removed or not) by their value. If you have other suggestion please let me know. 

Cheers,

Minh Duc

[Kato Richardson]

> since that user's id is no longer in the Members array of the group and they have no info

Store the user info separate from the group in this case or mark members removed instead of removing them completely.

>  if not implement a way of mass update when the master data change, then if the user change their name or avatar image multiple time while chatting, then it would appear in the chat to other user as multiple different users chatting.

There are indeed tradeoffs, but this works fine for gmail and other scalable apps of this sort and doesn't prove to be disruptive.

☼, Kato

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/135b2654-ab54-488a-a2a8-181bbac2a6ban%40googlegroups.com.

[Minh Duc Trinh]

Hey Kato,

Thank you for the reply. As mentioned in my previous email, I have changed the Members array into a Map to store the userId as key and a boolean as value to determine whether or not the user has left the group without losing the userId altogether. I also thought of storing the user info locally during the app runtime. Since I load 20 messages at a time whenever the user enter the chat or scroll up, I could check if there are id that not in the group chat and load the data once using it and apply the same for other message, but I have not apply this yet. For your second suggestion, it indeed works like that for Gmail and other scalable apps, but since I am making a real-time chat app, having user display differently during chat is not desirable or good for the User Experience. There is also the option of storing member data in a subcollection of a group document to avoid the size limit of a single document if the number of members in a group grow too big, but I suspect the scope of my app will not go that far. For now storing the Member data in a Map or an array of Map seems to be the solution for my current problem. If you have a better solution please let me know. Thank you for your help and suggestion.

Cheers,

Minh Duc

[Minh Duc Trinh]

Hello,

Want to open this up since I have another query on the data structure and how to process them to save the amount of read to Firestore. Currently in my Chat App I display message in each chat with an increment of 10, that's mean I display the 10 newest messages first and if the user scroll up it will display the previous 10 messages before that and so on. I also save the user data of the members of the Group locally the first time a user enter the chat so that when displaying a message it can also show the sender information along with it (like name and profile picture).

The members data is stored in an Array of Map as a field of a Group Document in Firestore, if a member leave the group their data is still store but their isActive field in the Map would change to false, indicating they have left the group and not shown in the group Members list for other user. Doing this can help display their info for their old message if other user read them. In a short term this would be fine, but in a long term, supposely if a group have added 1000 members but only have around 100 currently active member in the group, if a new user join or added to the group they would be pulling 1000 documents from Firestore. If multiple users do that then it will increase the number of read by a lot, and not all of them would check old messages from users who have left.

So I thought of only getting user information for members of the group from the messages that is currently display, and if an user scroll up and encounter a message that has user info not get and store locally, it will get them and store them locally so next time they won't need to. This could reduce the amount of read needed to display user info along with the messages but that also might result in longer load time to display previous message. I am not sure if this is a better way than what I currently have now so can anyone give me some pointer or recommendation?