BS Limits of Cloud Firestore - And the Inability to Horizontally Scale

[Kaan]

In this age, 1,000,000 concurrent connections is the dream, so let's ignore that

But, 10,000/s writes per second to the entire database - what kind of a funny limit is that?

I mean it's really agony having to make binding database decisions like this, and after making these decisions, discovering (or in my case, I assume re-discovering, I don't recall seeing this absurd limit when I first selected Cloud Firestore) these limits

For get 1,000,000 concurrent users, with 100,000 active users/players/clients-whatever - that 10,000/s limit just takes App Engine from infinitely scalable to a joke

We can't horizontally scale either, not possible to create more databases, but even if you could, the 10,000/s limit would need you to scale everything horizontally, even just authenticating the users etc.

Also there's the attack possibilities, just 2,000 people could come together, just make regular requests to your service, and they've essentially brought your entire project down

Is this something that will be removed/solved in the future, or did I do a dire mistake of choosing the Native mode?

[Kaan]

I probably originally read this article, and since it's always best to use new technologies, I picked the native mode: https://cloud.google.com/datastore/docs/firestore-or-datastore

In the above article, you can't see the 10,000/s limit, which is basically a MAJOR bottleneck, I mean, even in my 10 year old app, I do more than 10,000/s writes throughput-wise, with mapreduce-like tasks ...

[Kaan]

I guess if we could just use multiple databases, and also easily change the default database, this kind of heartbreak would be prevented

As I see, I made a mistake by choosing the Native option, my approach is to have scalability, and from experience, that 10,000/s limit just makes the native mode a good fit for highly agile / practical but small apps, rather than anything that processes data or scales

For example, one could create 10 firestore databases, and just use them to broadcast data to 10 million people, for that use case, it's very practical

But for an actual heart of a project, it will just result in a heart attack and early death

Since my databases are empty, I hope I can easily fix the mistake, I guess I'll open a ticket, maybe delete/re-create the apps etc.

[Kaan]

A bit constructive criticism, my solution approach would be

1) Make Datastore Mode default, don't confuse anyone

2) Make it easier to create and connect to other databases

3) Make it possible to change the default database (People could use Datastore Mode, create a Native one, migrate their data within their app, switch the Native to be the default, easy peasy, or they could just use them both)

4) (Comes with 2) Make sure new developers have a development database too, a live database they use for development, and maybe set the GOOGLE_APPLICATION_CREDENTIALS etc. automatically, or make things more practical

I currently created another project for development, and using it's Datastore mode for the time being - it's my step (4) - so it was needed anyway

(4) Goes into the territory of the currently "you're on your own" local development approach - just hold their hands, most integrations are there anyway, back in the day they were local, with more complicated products, it's obviously separate  live databases, buckets etc. to be used for development, that they can access locally (when online)

Still, it's really sad that I can't use a new product, maybe I'm autistic and I can't get over this fact, but is there really a database-wide limit of 10,000/s writes? If I commit to the old Datastore Mode, and if this limit gets removed in the future, I'll literally cry

[Elliott (Cloud Platform Support)]

Hello,

I understand that you are concerned about 10,000/s writes per second limit to the entire Cloud Firestore database. I was able to confirm this behavior here and I do apologize for the frustration this is causing you. All projects grow and because of this, limits will be reached as time goes on. I have consulted the information I have and found an opportunity provide you with an existing feature request but please note that there is no ETA no guarantee that it will be implemented.

You have suggested other improvements you wish to make and the best way forward is to use Google Issue Tracker to initiate the process.

Again, I know you put a lot of thought when you reached out to us. If you require any further assistance please let us know.

[Kaan]

Thanks, I guess this issue I created sums it up then: https://issuetracker.google.com/issues/177236402