How to prevent billing attacks?

908 views
Skip to first unread message

Steve Hull

unread,
Mar 15, 2020, 12:03:19 PM3/15/20
to Firebase Google Group
Hello!

As I'm working on a project that will likely leverage many pieces of Firebase infrastructure (though only Firestore & Auth to start), I keep having this nagging worry in the back of my head:

What if someone decides they don't like me (or one of my customers) and so they open console and run something like
 
while(true) { db.collection("posts").forEach(post => console.log(post)) }
 
And let that run for a few days. Assuming there are many posts in the "posts" collection, it seems like this could prove problematic from a billing standpoint very quickly.

For small apps I doubt this is a problem that would come up. But for anything ambitious, it seems like this is a show-stopper. Is there some sort of builtin protection against this sort of abuse?

Thanks,
Steve

Lander Beeuwsaert

unread,
Mar 15, 2020, 6:48:05 PM3/15/20
to Firebase Google Group
I actually have a related experience with that, admittedly, our own fault.
We made a wrong code change and suddenly all the users were in a loop, constantly updating a specific document.
I think we blew through 300.000 writes and some 1.000.000 reads in one hour, I can tell you, the moment we understood our error, it was all hands on deck :).
Would be nice if there is at least a sort of warning about this. Abnormal behaviour flag or something.

Jerry Sumpton

unread,
Mar 16, 2020, 11:11:24 AM3/16/20
to Firebase Google Group
Totally reasonable to have a daily budget and alerts at certain usage points. I like very much how Google provides that to me for my overall project budget.

Dan I

unread,
Mar 16, 2020, 11:55:01 AM3/16/20
to fireba...@googlegroups.com
That's fine but original question is - is there any security in place to stop such attacks. 

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/c7b7d6c4-a82e-4155-9d02-31defc372038%40googlegroups.com.

Sam Stern

unread,
Mar 16, 2020, 2:28:32 PM3/16/20
to Firebase Google Group
Hi all,

There is no built-in mechanism to stop such attacks, if you are concerned about this the best protection is to tighten your security rules.  One common method is to inspect the "request.query.limit" property to enforce that all queries contain a .limit(...) clause and that the limit size is reasonable.  This would prevent someone from just crawling the publicly visible parts of your DB without at least paging.

- Sam

Jerry Sumpton

unread,
Mar 17, 2020, 4:33:11 AM3/17/20
to fireba...@googlegroups.com
I am pleased to see that limit can be incorporated into rules. I didn’t know that, but I expect the “Sam’s” of our world would clear this hurtle like an Olympic athlete.

Firestore paging is relatively easy and effective. Works for me, and I expect that my site hackers are more skilled than I am.

I have committed to Firestore for my dream project, and I can’t now see how anyone in Twitter scale, or other large social site would expect to be able control their data. It looks like any site using Firestore basically is providing a firehose (pun) to anyone authenticated.

I am super concerned about this issue and fully expect to migrate this project to another database if we gain traction. The attractions are scale and dynamic data (whatever you call it). Firestore is amazing, but I have refactored all of our relationships to Redis, primarily because the performance for “hits” into user streams. For many queries, we now serve the entity Id and then populate the data an item at a time from Firestore.

A side note is that I am hosting a backend API on Node and Express using Cloud Run — which is pretty great. With an API, we have some options for state and limits, but I don’t yet know how.




Sam Stern

unread,
Mar 19, 2020, 2:27:21 PM3/19/20
to Firebase Google Group
Hi Jerry,

In the end you are absolutely right, if a hacker was determined to cause you pain by running up a large bill against a data set that they are allowed to access there's really no solution to stopping them.  In practice I have not heard from any customer experiencing an attack of this kind, I guess the hacker would have little to gain besides mayhem.

While I am not here to tell you that Firestore is the best database in the world for all apps, I don't think this discussion is a reason to abandon Firestore at scale.  Firestore supports secure access from client apps, nearly all other popular database require you to run an intermediate API server.  If you're willing to do that, you could put the same server in front of Firestore as well.  Even though you'd lose the realtime and the nice SDKs, you'd still have a multi-regional, fully-managed, strongly consistent and scalable database.  That's a pretty compelling package (imo) and it's why many large apps have been built on Cloud Datastore over the years.

In my experience working with developers, any sophisticated app has multiple data stores.  For example you can use Redis for your high-throughput KV data, Postgres for your reporting data set, and something like Firestore for synchronizing data to mobile apps across platforms.  And when you get to the scale of something like Twitter you probably have some data stores that you invented in house. 

Good luck building your app and please come back here with questions along the way!

- Sam

Steve Hull

unread,
Mar 20, 2020, 2:56:12 PM3/20/20
to Firebase Google Group
I imagine this is one of those "good problems" to have.

I think that I'll put a cap on my monthly bill and if someone tries to attack me this way my service would be briefly unavailable as I triage & maybe Google or the Firebase team would help me if it came to that.

> other popular database require you to run an intermediate API server

To be clear, I've built my fair share of app servers and rarely have had to deal DoS attacks. But for the past 7-8 years I've just put my API servers behind Cloudflare and figured "good enough" for D(D)oS protection. And frankly, the "attacks" we've dealt with were not really coordinated attacks but bot farms scraping content, and Cloudflare worked great for mitigating that.

> If you're willing to do that, you could put the same server in front of Firestore

This is a really good & interesting point. Do you think in my firebase config (using web / js sdk) I could point `databaseURL` to some domain I own (firebasedb.my-app.com) and use Cloudflare which would just CNAME to my-app.firebaseio.com?  I think this *would* work but if I "enable cloudflare" (by using the little cloud icon) it might break realtime like you said. Not sure how that works (websockets?) but even that might still work. I guess without cloudflare enabled the fact that firebasedb.my-app.com actually resolves to my-app.firebaseio.com would be public and a hacker could still attack me by making requests straight to my-app.firebaseio.com. So I'll need to test out enabling cloudflare

Haha I just found this posted on Reddit. Sounds like this will work beautifully. Cheers & thanks for responding everyone!

-S


Sam Stern

unread,
Mar 20, 2020, 3:32:07 PM3/20/20
to Firebase Google Group
Hi Steve,

Just to clarify I was not talking about putting a CDN directly in front of the database.  I've never tried that, your mileage may vary.  I meant something more like this:

// This endpoint could be a simple Node.js server that does this query and returns the results as JSON:
//    db.collection('items').where('category', '==, 'food').get()

With 99% of the databases on the market you have to do something like that.  You could do it with Firestore as well, and you'd end up with something with a much more "traditional" threat model.  Security and rate limiting would be up to you, and obviously you'd have to write all your own client code.

- Sam

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.

Tadej Dev

unread,
Mar 21, 2020, 9:43:13 AM3/21/20
to fireba...@googlegroups.com
Maybe something like "last access time" could be made available to firestore rules, so this way we can throttle requests on our own.

Similar to throttling update requests, which is already doable:
function canUpdate(){
return request.time < resource.data.updatedAt + duration.value(60, 's');
}

Dan I

unread,
Mar 21, 2020, 10:00:16 AM3/21/20
to fireba...@googlegroups.com
That's good idea, how about reads

John Sunchine

unread,
Mar 21, 2020, 11:17:23 AM3/21/20
to Firebase Google Group
I believe reads and writes are billed even if they are denied by the security rules (please correct me if I am wrong, I couldn't find the reference documentation I seem to recall when I just checked).

Given this, any sort of security rule based throttling won't work. Firestore really needs a DoS protection layer or additional capabilities to help manage this.
To unsubscribe from this group and stop receiving emails from it, send an email to fireba...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fireba...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fireba...@googlegroups.com.

Dan I

unread,
Mar 21, 2020, 11:25:56 AM3/21/20
to fireba...@googlegroups.com
My bad, it is billed as per the documentation. 

To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/e58732b9-55e3-464c-82af-dd4c70c14d8c%40googlegroups.com.

Steve Hull

unread,
Mar 22, 2020, 9:10:54 AM3/22/20
to Firebase Google Group
I don't think `last_access_time` would work because it would need to be per-user, or better yet, per-IP.

My domain is still transferring to Cloudflare but I'm holding out hope that it will work transparently.

-S

Tadej Dev

unread,
Mar 22, 2020, 12:43:32 PM3/22/20
to Firebase Google Group
@Dan I, as mentioned for reads firebase team would have to add support for that.

@John, didn't know one is billed for rejected requests. I guess this is something that firebase team could also change. Although I agree, Google's (DDOS) protection may be more user friendly (with rules I could imagine there are edge cases where user would be rejected where he shouldn't be).

@Steve, beside per-user, it may also need to be per-resource, so one could perform that check for each resource separately. Guess not the best solution.

Geoffrey Bourne

unread,
Mar 24, 2020, 6:51:42 PM3/24/20
to Firebase Google Group
I always put a CDN like CloudFlare in front of any Firebase hosting instance I have. Not only does it protect you from DDoS attacks, but saves you hits against your usage quota. Article I wrote on why have a CDN in front of Firebase.

But you're right, a hacker could try to screw you over by racking up a large bill. The best approach I have is set up the CDN with rate limiting. This will prevent you from having someone kill you in a single day. Then monitor daily if you have any large spikes in usage. You can do this either via the Firebase Console or Google Console. You unfortunately can't rely on the billing since that comes at the end of the month. Also, check out my app fireRun.io that monitors your daily Firebase usage, costs, and how much of your free tier you have left. Comes in a daily report. Might help ease your valid concerns.

Robert Grant

unread,
Nov 6, 2020, 9:47:55 AM11/6/20
to Firebase Google Group
Sorry for resurrecting this thread, but I had a further question. It look as though the (excellent) anothermadworld article on putting a CDN in front of Firebase is only going to prevent billing issues with static content hosted in Firebase Hosting. It's not going to protect actual database access. Is that right?

The places I'm imaging where these things can be at least a little addressed are:
1) CDN in front of static file hosting (check!)
2) Geo-IP restriction of firebase functions and firestore (maybe Google Cloud Armor?)
3) Requiring certain signup types only for users accessing firebase functions and firestore. I.e. so anonymous users can't access, or only if they've done bot detection. (I think this is doable with security rules, although this may run a rule that costs money every time.)
4) User and/or IP-based rate limiting access to firebase functions and firestore (perhaps CloudFlare or similar could buffer access to firebase functions?)

Apologies for the ramble! Hopefully these thoughts are useful for someone.


Rob

Sam Stern

unread,
Nov 6, 2020, 9:57:04 AM11/6/20
to Firebase Google Group
Hi Robert,

Not to say that anyone is wrong to do it, but we don't recommend putting a CDN in front of Firebase Hosting (which already has a CDN) or any other Firebase product.  Adding restrictions on your public HTTP functions is a good idea, as is making sure you don't allow Anonymous users to do more than the bare minimum you need.  As you know, Anonymous Auth users are not in any way verified.

While it's more reactive than defensive, here's a recent video we made about how to monitor your billing:

For what it's worth I personally have never seen an "attack" against a Firebase developer that was meant purely to run up a bill.  I have seen instances of an attacker trying to steal all the data in a database, which of course can be expensive for the app owner, but the target was the data not the app owner's cloud bill.  That's not to say a billing-based attack is not possible or it has never happened, just that I haven't seen it.

- Sam

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/b93ed7c8-a2da-452e-a089-378cd4e3daecn%40googlegroups.com.

Robert Grant

unread,
Nov 10, 2020, 9:36:16 AM11/10/20
to Firebase Google Group
Hi Sam

Thanks so much for this - that's really detailed feedback! 

My approach is currently to do some writes using firebase functions, particularly the ones setting up access levels to the data, and reads and fast-paced, synchronisation-friendly writes using the Firebase sync feature. I'm doing that because I don't trust my ability with the security rules enough, and that simplifies things for me. It just feels easier to rate limit firebase functions, for some reason.

I'll see how it goes, and will feed back for all your delight or delicious schadenfreude.


Rob

wekaso...@gmail.com

unread,
Nov 10, 2020, 9:39:53 PM11/10/20
to Firebase Google Group
Steve commented: "I think that I'll put a cap on my monthly bill" 

I haven't watched the video Sam posted yet (its been on my list)  but as far as I am aware there is no longer any way to cap billing only to set budget alerts.
So if someone did want to rack up a large bill then you would want to have your budget alerts on and simply hope you catch them in time then take down your platform. 

Robert Grant

unread,
Dec 10, 2020, 11:17:27 AM12/10/20
to Firebase Google Group
@sam this is the sort of thing that's concerning: https://www.theregister.com/2020/12/10/google_cloud_over_run/

Sam Stern

unread,
Dec 10, 2020, 11:52:29 AM12/10/20
to Firebase Google Group
Hi Robert,

I completely agree with you, that situation is very scary and is not properly addressed by any of our current billing solutions. Reading that article is yet another reminder of why we need to focus on improving our billing controls. As mentioned at the end of the article we gave the developer a refund for the unexpected charges, so I hope you can at least see that we don't want to make money from mistakes.

- Sam

Lahiru Chandima

unread,
Dec 11, 2020, 12:14:20 AM12/11/20
to Firebase Google Group
@sam it would be a relieve to at least hear that you guys are working on improving billing controls. It appears that automatically shutting down the firebase project on a billing alert wouldn't work, if there is a delay in updating billing as mentioned in that post.

Steve Hull

unread,
Dec 15, 2020, 5:30:56 AM12/15/20
to Firebase Google Group
> As mentioned at the end of the article we gave the developer a refund for the unexpected charges, so I hope you can at least see that we don't want to make money from mistakes.

That's all well and good but unless it's spelled out as an official policy and ideally in ToS, one can hardly rely on goodwill from Google (or any other company for that matter). A cynical observer might say "yes Google refunded the bills for this (and one or two other well-known incidents) but how many have been forced to pay outsized bills? Maybe the only way one can hope for mercy is to write a post that receives many views."

-S

Kato Richardson

unread,
Dec 15, 2020, 5:16:53 PM12/15/20
to Firebase Google Group
Hi Steve,

> but how many have been forced to pay outsized bills?... Maybe the only way one can hope for mercy is to write a post that receives many views.
As the author of the Firebase goodwill refund policy (yep, that's the name of it), I can say that we are lenient in regards to coding mistakes and dramatic usage spikes. That's not in the terms of service and could change some day as you've pointed out. But you don't need to write a Blog post of any kind in order to request a one-time reprieve. You can just reach out to support.

> it would be a relieve to at least hear that you guys are working on improving billing controls
There is an ad hoc solution in the interim; Todd Kerpelman just finished the last release in a series about billing caps he's been working on for some time. You can check it out here.

☼, Kato






--

Kato Richardson | Developer Programs Eng | kato...@google.com | 775-235-8398

wekaso...@gmail.com

unread,
Dec 16, 2020, 5:37:13 AM12/16/20
to Firebase Google Group
It looks like there is a programmatic way to put a hard cap on billing now.  

Flon Mackenzie

unread,
Dec 16, 2020, 5:37:21 AM12/16/20
to 'Sam Stern' via Firebase Google Group
It looks like there is a programmatic way to put a hard cap on billing now.  
You received this message because you are subscribed to a topic in the Google Groups "Firebase Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/firebase-talk/Fuvnk5mPoSU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to firebase-tal...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/96dbc8e7-7315-404c-9a60-1aa015535273n%40googlegroups.com.

Firebase User

unread,
May 12, 2022, 10:31:16 AMMay 12
to Firebase Google Group
Have you found a solution to this? We have a lot of possible bad actors that might write a simple infinity loop to make millions of requests to our Firebase Realtime Database and create thousands of dollars cost to us.

App Check doesn't protect from it either. I tried to spam the app and I was able to do it very easily from the Chrome console.

Has anyone found a solution to this?

Jean Omberto

unread,
May 12, 2022, 3:50:04 PMMay 12
to fireba...@googlegroups.com
Hi,
Just curious, 
Is the database protected by rules ?
If Only authorised users can read and write, you could write a script to verify if a user has sent x amount of requests in x amount of minutes or seconds.

If yes, 
Idea.
Store their uid in a collection of banned users so u can check against it any time in future.

On their device,  as soon as the next request is done,  redirect them to some other page.
On their localstorage, keep a key and value that identify them so that next time, this user doesn't send any requests to your server.

Localstorage.set (k, v)
Before any request is sent,  Verify the value of this key if it exists.

I rely on firestore rules, app check and this little idea above.
Not a magic bullet proof but it helps.

Regards,
Jean O.




--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.

Robert Grant

unread,
May 14, 2022, 10:38:02 AMMay 14
to fireba...@googlegroups.com
This is a good idea, but I believe that even that check requires a (billable) Firebase read.

You received this message because you are subscribed to a topic in the Google Groups "Firebase Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/firebase-talk/Fuvnk5mPoSU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to firebase-tal...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CACP%2B_1gC%2Bfs3YANWLRtXbkRh9Dr-W67rRprY5iC06-pUft8Stg%40mail.gmail.com.

Steve Hull

unread,
May 18, 2022, 5:30:35 PMMay 18
to fireba...@googlegroups.com
And doesn’t help at all for protecting resources that have public read. Most apps have some concept of allowing anonymous users to browse.

And the idea of using localstorage is great for doing local throttling to avoid accidentally incurring additional costs, but doesn’t address the “bad actor” concern. They’re just going to be hitting your firestore endpoints directly, either with a browser console or even just curl.

-S

Bryan Price-McConnahea

unread,
May 19, 2022, 1:03:00 PMMay 19
to Firebase Google Group
You shouldn't expose Firebase directly to the client, so no user should be able to access a resource via the console like in your original example code.  You should place calls to your Firebase DB behind an API, for instance a cloud function, and finally you should implement caching such that if someone were to attempt to repeatedly call your endpoint, all they would get back is cached data.  These steps would essentially make it impossible for someone without direct access to your server to run up a bill as you describe.

Laurent Payot

unread,
May 29, 2022, 6:11:29 PMMay 29
to Firebase Google Group
Being used on the client is the whole point of Firebase non-lite versions (offline capabilities, optimistic updates etc.). But even if used behind a cloud function, how to prevent billing attacks of cloud functions calls then?

Carsten Hagemann

unread,
Jun 2, 2022, 10:29:22 AMJun 2
to Firebase Google Group
Have you had a look at App Check? It looks to me that it could help you with that.
> App Check helps protect your backend resources from abuse, such as billing fraud and phishing. It works with both Firebase services and your own backends to keep your resources safe.

Reply all
Reply to author
Forward
0 new messages