Limit the collection write rate

[Kane Kim]

Hello,


While browsing best practices I've found:

Limit the collection write rate
1,000 operations/second

Keep the rate of write operations for an individual collection under 1,000 operations/second.

What does it mean? Does it imply we can't write to collection (utilizing all 10k write limit) while someone is listening on it?

Do we have to shard collections now (this is very inconvenient).

Thanks.

[Kane Kim]

It is unclear what is the root of this limitation, will it be different if subscriber is listening on group collection (across subcollections)?

[Kato Richardson]

Hi Kane, 

Are we talking about Firestore or Realtime Database? Performance characteristics and thus the answer will vary significantly.

☼, Kato

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/73cf1392-560a-436c-8bb7-ad5484bb2374%40googlegroups.com.

--

Kato Richardson |

Developer Programs Eng |

kato...@google.com |

775-235-8398

[Kane Kim]

Thanks for getting back to me! It's firestore.

On Monday, May 4, 2020 at 11:25:15 AM UTC-7, Kato Richardson wrote:

Hi Kane, 

Are we talking about Firestore or Realtime Database? Performance characteristics and thus the answer will vary significantly.

☼, Kato

On Fri, May 1, 2020 at 4:10 PM Kane Kim <kane....@gmail.com> wrote:

It is unclear what is the root of this limitation, will it be different if subscriber is listening on group collection (across subcollections)?

On Friday, May 1, 2020 at 10:26:37 AM UTC-7, Kane Kim wrote:

Hello,


While browsing best practices I've found:

Limit the collection write rate
1,000 operations/second

Keep the rate of write operations for an individual collection under 1,000 operations/second.

What does it mean? Does it imply we can't write to collection (utilizing all 10k write limit) while someone is listening on it?

Do we have to shard collections now (this is very inconvenient).

Thanks.

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to fireba...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/73cf1392-560a-436c-8bb7-ad5484bb2374%40googlegroups.com.

[Kato Richardson]

Thanks Kane; I see this in the Firestore best practices, but I'm not sure why it's there. We'll look into it and post back here.

To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/ced5921a-a603-4531-aba2-71b507ed1845%40googlegroups.com.

[Gil Gilbert]

Hi Kane,

The 1000 writes per second to a single collection is not a hard limit. There's no quota for this. The issue is that notification latency for new queries against a collection goes up as the level of sustained writes increases. The 1000/second number is a guideline for a level of writing where we are confident that you can continue to see low latency for query results.

If there are no listeners to a collection or you're temporarily not interested in notification latency, you can go higher. For example, if you're bulk loading data, there's no reason to limit yourself to 1000 writes/second.

Also note that bursts of writes that happen to temporarily go over this aren't really a problem--it's sustained workloads over longer durations where you'll want to avoid high write rates.

Cheers,

-Gil

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/ced5921a-a603-4531-aba2-71b507ed1845%40googlegroups.com.

[Kane Kim]

Gil, thanks for the explanation.

Does it apply also to listening across collection groups?

If listener is subscribed to collection group can summary write throughput be higher than 1000/second and still provide low latency?

[Marek Gilbert]

If you have listeners to a collection group then this does apply. If there are no listeners to a collection group then each collection is independent.

You received this message because you are subscribed to a topic in the Google Groups "Firebase Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/firebase-talk/PWanrNOqimc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to firebase-tal...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/c9d768fa-ec9a-49d1-8669-db9e44aa5885%40googlegroups.com.

[Kane Kim]

My use case for this is:

I have a structure similar to chat app described in https://cloud.google.com/firestore/docs/concepts/structure-data.

In addition to that I am storing last message timestamp in a room document. Updates to a single room are limited (not very often), but there may be more than 1000 rooms that I am subscribing to, so total number of writes will most likely exceed 1000/second limit (e.g. 1 room has only 1 update per second, but there are 10000 rooms). What would be the best way to structure this?

On Tuesday, May 5, 2020 at 9:51:31 AM UTC-7, Gil Gilbert wrote:

[Kane Kim]

Also forgot to mention, obviously listeners subscribe only on subset of those rooms (not all 10k). E.g. listener1 might listen only on 10 rooms, filtered by some condition, listener2 to other 10, etc).

So in this situation total write rate to a collection could be 10k, but subset that particular listener sees is pretty small.

[Marek Gilbert]

The limitation that the 1000 writes/second guideline is dealing with applies to the collection as a whole. It doesn't matter whether listeners listen to the whole collection or not or whether their queries are overlapping.

It's hard to make a recommendation without knowing how you plan to use the timestamp in the room document. Another factor is how sensitive to latency you are for queries against the rooms collection. Another question to answer is whether or not approximate timestamps are acceptable.

For example, if approximate timestamps are acceptable, then you could only update the timestamp in the room if it hasn't been updated in a while, or apply a rate limit to such updates.

Alternatively, you could update the room timestamps in a Firebase Function and apply a rate limit/batching there.

Depending on how you're using the timestamps, you might combine batching/rate limiting with a binning strategy wherein you put multiple room updates into a single document. If you're just using this as a trigger for clients to go and fetch updated messages, this might be an acceptable way to bring the write rate down.

Another option is to keep the timestamp in a subcollection of the room (and not keep a room document at all). This reduces the ability to query across rooms to find out which were updated last, but depending on your application it might be acceptable to just query the rooms of interest directly to see if they've changed.

A final option is to keep a timestamp per user rather than per room. You can store conversation updated timestamps along with other high velocity data in a separate collection per user. Doing this kind of fan out in a Function works particularly well and is the approach I'd probably take.

Finally note that if you're planning on indexing the timestamp (so as to be able to query for the most recently updated room) you're going to create a hotspot in your index and will run up against the 500 writes/second limit when indexed fields are consecutive or have similar values. This is really the principle problem to solve (though the mechanisms/discussion are the same).

Good luck!

-Gil

You received this message because you are subscribed to a topic in the Google Groups "Firebase Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/firebase-talk/PWanrNOqimc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to firebase-tal...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/e3110076-0263-475a-98d6-83d48844174b%40googlegroups.com.

[Kane Kim]

On Tuesday, May 5, 2020 at 1:24:06 PM UTC-7, Gil Gilbert wrote:

The limitation that the 1000 writes/second guideline is dealing with applies to the collection as a whole. It doesn't matter whether listeners listen to the whole collection or not or whether their queries are overlapping.

Ok, thanks, this is useful to know. Does it apply to listening to collection groups? As I understand in this case writes will be spread across subcollections?

It's hard to make a recommendation without knowing how you plan to use the timestamp in the room document. Another factor is how sensitive to latency you are for queries against the rooms collection. Another question to answer is whether or not approximate timestamps are acceptable.

For example, if approximate timestamps are acceptable, then you could only update the timestamp in the room if it hasn't been updated in a while, or apply a rate limit

to such updates.

Yes, I am already planning to apply rate limit, and anyway, updates to single document are limited to 1/sec, so I can't update it very often anyway.

 

Alternatively, you could update the room timestamps in a Firebase Function and apply a rate limit/batching there.

Depending on how you're using the timestamps, you might combine batching/rate limiting with a binning strategy wherein you put multiple room updates into a single document. If you're just using this as a trigger for clients to go and fetch updated messages, this might be an acceptable way to bring the write rate down.

This sounds interesting, I will think along these lines.

Another option is to keep the timestamp in a subcollection of the room (and not keep a room document at all). This reduces the ability to query across rooms to find out which were updated last, but depending on your application it might be acceptable to just query the rooms of interest directly to see if they've changed.

And again, if I keep timestamp in subcollection and listener listens on collection group - this will still suffer from same problem? (I assume so, just want to make sure).

 

A final option is to keep a timestamp per user rather than per room. You can store conversation updated timestamps along with other high velocity data in a separate collection per user. Doing this kind of fan out in a Function works particularly well and is the approach I'd probably take.

Fanning out writes might work, but firestore has a hard write limit that I would like to extend as much as possible, fanning out writes will quickly multiple the writes by orders of magnitude.

Finally note that if you're planning on indexing the timestamp (so as to be able to query for the most recently updated room) you're going to create a hotspot in your index and will run up against the 500 writes/second limit when indexed fields are consecutive or have similar values. This is really the principle problem to solve (though the mechanisms/discussion are the same).

Yes, I am aware about it, I am disabling default timestamp index and creating a composite index combined with filter condition as a prefix and then timestamp so hotspot is within the narrow scope. So the only part that doesn't scale for me is ability to listen and be notified about updates on this big collection.

I can't really subscribe directly to the documents I am interested in (as condition might include too many of them) and firestore obviously doesn't support joins.