Cloud Firestore Write limit

[Sachin]

Hi All,

Currently we are migrating from real time database to Cloud Firestore.  In our application, we write approximately 20 message per sec. 

As per Firestore limits, 

Maximum write rate to a collection in which documents contain sequential values in an indexed field

500 per second

To understand this, I have written a sample code which uploads following JSON with one field i.e. PacketGenTimestamp

{

     PacketGenTimestamp: Milliseconds since Unix Epoch,

};

And the structure of collection is

test (collection name)-> dataX (document) -> PacketGenTimestamp(Field)=milliseconds(Time)

where X is 1,2,3,..............................

I am calculating time required to send the message as below - 

        firebaseCollection.doc("data"+index).set(msg)

        .then(function() {

            console.log("Success -> " + (GetTimestamp() - msg.PacketGenTimestamp));

        })

        .catch(function(error) {

            console.error("Error -> ", error);

        });

Following are the observations - 

1. No of messages per sec  < 10

    Time required to send message is around 100 ms

2. No of messages per sec  > 10 

    Time required to send message is increases monotonically to seconds and never comes down below 1 sec.

If limit is 500 per seconds, why I am observing this behavior?

Thanks,

Sachin

[Aaron Phillips]

Hi Sachin,

We encountered a similar issue, I think you might be suffering from

https://cloud.google.com/firestore/docs/best-practices#hotspots

Might be worth letting Firestore generate the document id to avoid the problem.  We often put our ID as a field so we can at least query for documents using incremental ids.

Aaron

[Jerry Sumpton]

How is this handled with batch? For example, a batch loop might increment a property multiple times within the batch size and then commit. Will the writes be consolidated, or sequential?

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/615e9f58-e9a0-4323-ae8a-84eaa07b14a9%40googlegroups.com.

[Kato Richardson]

Hi Jerry,

It looks like the Firestore server will batch the writes to a single document at the hardware level to help with performance, but this doesn't affect the billable writes. So if this is high volume, some consolidation on the client could be appropriate. Note also that if the field being updated is not indexed, you should achieve a higher throughput rate since the index is the main bottleneck for sequential values. And in case it's unclear, that doesn't apply to the OP's questions, since those were about document keys and adds, which are slightly different than updates to a doc through batches.

I hope that helps!

☼, Kato

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/5A7D0AF9-484D-41FD-9564-CCDA9E07C86F%40gmail.com.

--

Kato Richardson |

Developer Programs Eng |

kato...@google.com |

775-235-8398

[Jerry Sumpton]

The “don’t index” tip is a good one. I currently don’t require a sort by any “counter” properties. I would not have thought about this without you pointing it out.

This is primarily where I am deleting an entity with many associated things like images, likes, follows, comments, and whatever and I delete it. The lazy way is to iterate each associated thing and undo it — forEach > unlike as an example. The “like” side is required for each, but the counter could be skipped. It is just more code and overhead to deal with, and almost meaningless in Redis — which I just migrated from.

That is likely a bit confusing. I do appreciate your tips.

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CADypTEYO3ux2uZfknGxsotRMk%3DauShrONu0QB0gK%2BhYvtmfBWw%40mail.gmail.com.

[Sachin]

Hi

Thanks for reply. 

I changed the code as you suggested i.e. 

    var msg = {

        PacketGenTimestamp: new Date().getTime(),

    };

    firebaseCollection.add(msg)

        .then(function() {

        console.log("Success -> " + (new Date().getTime() - msg.PacketGenTimestamp));

    })

    .catch(function(error) {

        console.error("Error -> ", error);

    });

But results are the same

Example - If I send 20 messages per sec, result is 

firebase.js:142 Success -> 408

firebase.js:142 Success -> 504

firebase.js:142 Success -> 581

firebase.js:142 Success -> 605

firebase.js:142 Success -> 677

firebase.js:142 Success -> 742

firebase.js:142 Success -> 813

firebase.js:142 Success -> 863

firebase.js:142 Success -> 927

firebase.js:142 Success -> 951

firebase.js:142 Success -> 969

firebase.js:142 Success -> 987

firebase.js:142 Success -> 1047

firebase.js:142 Success -> 1108

firebase.js:142 Success -> 1186

firebase.js:142 Success -> 1227

firebase.js:142 Success -> 1274

firebase.js:142 Success -> 1364

firebase.js:142 Success -> 1394

firebase.js:142 Success -> 1431

firebase.js:142 Success -> 1452

firebase.js:142 Success -> 1513

firebase.js:142 Success -> 1605

firebase.js:142 Success -> 1629

firebase.js:142 Success -> 1651

firebase.js:142 Success -> 1667

firebase.js:142 Success -> 1712

firebase.js:142 Success -> 1807

firebase.js:142 Success -> 1811

firebase.js:142 Success -> 1852

firebase.js:142 Success -> 1870

firebase.js:142 Success -> 1893

firebase.js:142 Success -> 1910

firebase.js:142 Success -> 1943

firebase.js:142 Success -> 1965

firebase.js:142 Success -> 2049

firebase.js:142 Success -> 2122

firebase.js:142 Success -> 2208

If I send 5 messages per sec 

firebase.js:142 Success -> 78

firebase.js:142 Success -> 79

firebase.js:142 Success -> 84

2firebase.js:142 Success -> 79

firebase.js:142 Success -> 84

firebase.js:142 Success -> 82

firebase.js:142 Success -> 79

firebase.js:142 Success -> 80

firebase.js:142 Success -> 79

firebase.js:142 Success -> 145

Let me know what I am missing here. I also checked with index exemption for timestamp field.

Thanks,

Sachin

[Kato Richardson]

Sachin, any chance you could put together a small code sample that reproduces the behavior so we can tinker?

☼, Kato

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/004ea17f-6ef1-4a65-83aa-7e03a261405c%40googlegroups.com.

[Sachin]

Hi

Please find attached sample code i.e. HTML and JavaScript file.  

Steps - 

1. Rename attached file i.e. remove txt extension.

2. Open Send.html and add Firebase account details i.e.

    "apiKey": "XXXXX", "authDomain": "XXXXX", "databaseURL": "XXXXX",  "projectId": "XXXXX",  "storageBucket": "XXXXX",  "messagingSenderId": "XXXXX"

3. Deploy the files

4. Open the send.html in browser and open console.

5. Press Start button. It will start sending data.

6. Observe on console the required to send the message. 

You can change the message rate in send.html. Default setting is 20 messages per sec. 

Thanks,

Sachin

To unsubscribe from this group and stop receiving emails from it, send an email to fireba...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/004ea17f-6ef1-4a65-83aa-7e03a261405c%40googlegroups.com.