Questions about scaling / reliability of cloud function triggers

[Alex N]

I have some questions about reliability of firebase cloud function triggers at scale. My application sometimes needs to trigger thousands of cloud functions that all must be completed successfully.

• Is there a limit for how many cloud functions can be triggered by a single update or within a short period of time? I’ve found when I get over 1000 or so updates at once, none of the cloud functions are triggered.
• Sometimes functions are killed by the system with the message “Function killed. Error: system encountered unexpected error”. Are these retried? Is there any way to be notified of this so they can be retried?
• When are functions retried normally, how many times, and at what intervals? The docs indicate that if a function times out, it will be retried, though I haven’t seen that happen in any cases.

Thanks

[Tom Larkworthy]

Yes there are limits of the number of triggers you can transmit in bursts from the Realtime Database. It protects against accidentally setting a trigger too low in the data hierarchy and consuming a ton of compute resources accidentally. It also protects connected systems from sudden massive spikes in traffic. I don't think that very large write magnifications are a feature we want to support as it has too many potential risks.

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.
To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/5531a506-830e-4aca-b6d5-e37d441446a2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nick Okunew]

Are these limitations documented? I understand too many functions from a single update, but limiting by time window etc effectively bounds the size of a solution that can work on the platform in this way.

[Tom Larkworthy]

So I am being a bit evasive about stating exact limits, becuase it's a beta product and we still want to tune them according to what makes sense for typical usage. I think the limits will probably be lowered from what they are now. We will document it then. If you are firing a small number of triggers per write (<20), you will run into database write scaling limits before triggers. Only in the case of trying to fire n triggers per write for larger n will you risk see missing triggers from rate limiting. 

On Apr 8, 2017 5:38 PM, "Nick Okunew" <naok...@gmail.com> wrote:

Are these limitations documented? I understand too many functions from a single update, but limiting by time window etc effectively bounds the size of a solution that can work on the platform in this way.

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.
To post to this group, send email to fireba...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/50d4746f-e8db-4011-b2bb-0d6896205ce4%40googlegroups.com.

[Ryan Mills]

HI Tom--Can you provide more information on Nick's questions "Sometimes functions are killed by the system with the message “Function killed. Error: system encountered unexpected error”. Are these retried? Is there any way to be notified of this so they can be retried?

When are functions retried normally, how many times, and at what intervals? The docs indicate that if a function times out, it will be retried, though I haven’t seen that happen in any cases."

Ryan

On Sunday, April 9, 2017 at 11:47:17 AM UTC+10, Tom Larkworthy wrote:

So I am being a bit evasive about stating exact limits, becuase it's a beta product and we still want to tune them according to what makes sense for typical usage. I think the limits will probably be lowered from what they are now. We will document it then. If you are firing a small number of triggers per write (<20), you will run into database write scaling limits before triggers. Only in the case of trying to fire n triggers per write for larger n will you risk see missing triggers from rate limiting. 

On Apr 8, 2017 5:38 PM, "Nick Okunew" <naok...@gmail.com> wrote:

Are these limitations documented? I understand too many functions from a single update, but limiting by time window etc effectively bounds the size of a solution that can work on the platform in this way.

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.

[Tom Larkworthy]

So the unexpected errors are likely bugs in the event processing. Could you file bug reports for them so we can investigate and ultimately make them disappear. We do not retry those types of failures. We keep logs for 7 days so if you can file a bug as soon as possible we have a better chance of diagnosing the underlying issue. 

To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/c79b21da-5c15-4a25-8172-076262ccda77%40googlegroups.com.

[Brendan Lim]

Hey Alex,

I'm interested in hearing more about your specific use case. Can you provide more detail about what you're trying to accomplish in your app which would require over 1000 triggers to be fired at once?

Thanks!

[Ryan Mills]

Hey Tom/Brendan--I'm also interested to know under what circumstances a function will and won't retry.

Ryan

[Alex N]

In this case, I'm filling in a massive grid where each cell requires an http request and some calculation. For a simplified explanation, I currently use the creation of each position in the grid to trigger this.

Obviously I should be throttling this anyway, but went without throttling during the initial development and tests. When I inserted over 1000 or so rows in one call, the system failed silently to trigger the cloud functions. I do understand why Google may not want to support this, though it's important to know the limits and be informed of the failure.

[Randy Sugianto]

I just encountered this behaviour today.

I wrote a cloud function to send a broadcast message to ~5000 chat users by writing to a certain path an object that has 5000 children. I suppose that will trigger the cloud function on that path. The function is not triggered at all. When I reduce the number of children to 996, the function is triggered. When I increase it again to 1071, the function is not triggered at all. I think there is a hard limit of 1000.

This should really be documented. The use case above is valid. We do not have a means to schedule calling functions yet, so the above scenario should be supported, or at least show an error.

[Alex N]

Randy, easy workaround is to _.chunk writes. I ended up doing the post-processing in the write function, which is much faster anyhow.

[Randy Sugianto]

In the end I wrote the changes in chunks where each chunk causes 100 function invocations.

However, every time I write about 55 chunks (each will trigger 100 function invocations), there are about 5-10 times where the function is not invoked successfully.

Unfortunately we do not have a retry mechanism yet. Are we going to have retrying soon?