Realtime Database - Massive spike in "Load" Usage?

[dr...@basedata.solutions]

Hi, I'm curious what effects the "Load" on the realtime databse?

The popover says:

"The percentage of your database that's busy processing requests (either realtime or RESTful) over 1-minute intervals. You may see performance issues as you approach 100%."

But I find some of these numbers unrealistic.  Normal values hover around 2%, yet on Aug 5, it reaches over 1000%.  None of the other simple metrics correlate with any spikes - downloads were a normal 500MB for the day.  I have nothing fancy going on.  No back-end processes running.  The chatter between app and DB is minimal.

Is there anything one can do to figure out what's going on here, and whether there are scale / architecture issues?

Thanks.

Drew

[Tyler Rockwood]

The Load stat is correct, although this is bug we haven't yet fixed. We measure this after each operation, and as the tooltip says it's over one minute intervals, what happened here is that some operation took 10 minutes, so you see a 1000%. 10 minutes is near our limits for how long a query can run, so it's likely a query took a long time and was eventually completed or cancelled by the database.

-Tyler

[dr...@basedata.solutions]

Thanks Tyler.  I appreciate the follow-up.

I don't have any queries in my app - it's always pulling keys directly.

I contacted support and there was an outage on that day: https://status.firebase.google.com/incident/Realtime%20Database/17070

It was queued up writes once the outage was resolved.  Probably due to the massive load of everyone trying to write at the same time, each op reported a massive latency.

From my perspective, this metric isn't very helpful as is.  Unless I notice this spike w/i 24 hours, there's no way to dig into and try and correlate the cause of a performance issue.  I'd love to see this metric at a more granular level, allow for alerts, and overlay outages / annotations.  This would allow me to correlate performance issues w/ deployments, marketing campaigns, outages etc and take real action to resolve.

Keep up the awesome work!  I've seen massive improvements over the last year.  I'm very excited for what's next!

Drew

[Tyler Rockwood]

We're working on it, stay tuned!

-Tyler

[Tyler Rockwood]

A long time coming, but we just released an integration with Google Stackdriver that will help here, for example we have a graph of your Load broken down by operation type.

Docs here: https://firebase.google.com/docs/database/usage/monitor-performance