FCM v1: QUOTA_EXCEEDED when sending to topics (not when subscribing)

[Eran Dagan]

I've recently migrated my app from the legacy batch sending API to FCM v1 topics. The app allows users to follow TV shows and get notifications when episodes air. Each TV show is a topic (say, "show-xyz"), alongside a general "toggle" topic for types of notifications the user is interested in (like "release-dates", "new-episodes"). So, a send request looks something like this:

{"payload":{"message":{"condition":"'show-24402' in topics && 'shows-airing' in topics","notification":{"title":"My Title","body":"S2024E138"}}}}

The problem is, after just a few submissions (with very generous backoffs), we're getting QUOTA_EXCEEDED from FCM. Some submissions target ~10K users, while others very few users. We never get close to the 600K messages limit (we stay around ~2000).

Here are example logs. Note the time between the "Sending message" and "Message sent" logs which is caused by rapid 429s. 

2024-07-12T19:15:01.382620Z - INFO - production.INFO: Sending message {"payload":{"message":{"condition":"'show-4310' in topics && 'shows-airing--50' in topics","notification":"...."}}} []

2024-07-12T19:15:02.339667Z - INFO - production.INFO: Message sent {"payload":{"message":{"condition":"'show-4310' in topics && 'shows-airing--50' in topics","notification":"...."}}} []

2024-07-12T19:15:02.362507Z - INFO - production.INFO: Sending message {"payload":{"message":{"condition":"'show-7197' in topics && 'shows-airing-40' in topics","notification":"...."}}} []

2024-07-12T19:16:03.208658Z - INFO - production.INFO: Message sent {"payload":{"message":{"condition":"'show-7197' in topics && 'shows-airing-40' in topics","notification":"...."}}} []

2024-07-12T19:16:03.214590Z - INFO - production.INFO: Sending message {"payload":{"message":{"condition":"'show-9255' in topics && 'shows-airing--20' in topics","notification":"...."}}} []

2024-07-12T19:16:03.668853Z - INFO - production.INFO: Message sent {"payload":{"message":{"condition":"'show-9255' in topics && 'shows-airing--20' in topics","notification":"...."}}} []

2024-07-12T19:16:03.684779Z - INFO - production.INFO: Sending message {"payload":{"message":{"condition":"'show-10640' in topics && 'shows-airing-40' in topics","notification":"...."}}} []

2024-07-12T19:18:04.635770Z - INFO - production.INFO: Message sent {"payload":{"message":{"condition":"'show-10640' in topics && 'shows-airing-40' in topics","notification":"...."}}} []

2024-07-12T19:18:04.641271Z - INFO - production.INFO: Sending message {"payload":{"message":{"condition":"'show-18256' in topics && 'shows-airing-40' in topics","notification":"...."}}} []

2024-07-12T19:18:05.069938Z - INFO - production.INFO: Message sent {"payload":{"message":{"condition":"'show-18256' in topics && 'shows-airing-40' in topics","notification":"...."}}} []

2024-07-12T19:18:05.074314Z - INFO - production.INFO: Sending message {"payload":{"message":{"condition":"'show-19275' in topics && 'shows-airing--20' in topics","notification":"...."}}} []

2024-07-12T19:19:05.658275Z - INFO - production.INFO: Message sent {"payload":{"message":{"condition":"'show-19275' in topics && 'shows-airing--20' in topics","notification":"...."}}} []

2024-07-12T19:19:05.664380Z - INFO - production.INFO: Sending message {"payload":{"message":{"condition":"'show-24402' in topics && 'shows-airing--20' in topics","notification":"...."}}} []

2024-07-12T19:22:06.888527Z - ERROR - production.ERROR: Failed to send message Client error: `POST https://fcm.googleapis.com/v1/projects/hobi-1/messages:send` resulted in a `429 Too Many Requests` response:\\n{\\n \\\"error\\\": {\\n \\\"code\\\": 429,\\n \\\"message\\\": \\\"Topic quota exceeded.\\\",\\n \\\"status\\\": \\\"RESOURCE_EXHAUSTED\\\",\\n \\\"details (truncated...)\\n at 

This makes working with FCM topics almost impossible - sending notifications to many different topics can take hours even though the actual number of messages that need to be sent is very small.

I couldn't find any documentation about this strange behavior, seems like an extremely low rate limit for a function as critical as topics.

I'd be grateful for any pointers. I don't want to migrate off topics since sending hundreds of thousands individual HTTP requests (even with HTTP2 multiplexing) is very error prone (and beats the entire point of using topics)

Eran