Deploying 150+ functions is very slow

[Parker Wightman]

Hi! We run a large web app whose backend is hosted on Firebase Functions. We have over 150 functions.

I understand that the architecture of cloud functions is such that each function is updated individually, but this can be quite slow when you have a lot of functions.

Due to rate limits, you can only update a handful (~10) at a time, so deploying all our functions takes ~45 minutes and growing for each function we add.

It's especially unfortunate since we're really just uploading the same bundle to every single function.

Are there any plans to improve this, that anyone is aware of? Where perhaps a single bundle could be uploaded to many functions at once?

We can, of course, consider consolidating how many functions we use (e.g. dispatching to different code paths ourselves within a single cloud function), but just curious if this is a known pain point for others that Firebase/Google Cloud might be looking into improving.

[Kato Richardson]

Hi Parker,

Would help to understand the problem you're trying to solve by deploying 150 functions. For example, if you are uploading the same bundle to each function, could you not parameterize the requests and just have one function to deploy? You mention something along this line, but it's unclear why that's not a better option.

Assuming this isn't the initial upload, you can of course deploy only the functions that have changed with the --only options.

☼, Kato

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/a72a9525-f763-4a98-90cb-a357e607d25en%40googlegroups.com.

--

Kato Richardson |

Developer Programs Eng |

kato...@google.com |

775-235-8398

[Parker Wightman]

We're not trying to solve a specific problem by deploying 150 functions, it's just how many we have, given that there's roughly 1:1 mapping of URLs -> functions. We do already consolidate and internally dispatch some endpoints that are REST-y in nature, but we just have a large app ¯\_(ツ)_/¯.

We do try to deploy independent functions when possible, but many times it's a change in functionality used across many functions so it becomes error prone to try to guess which ones really need deploying.

For scaling/cost reasons it's very useful to be able to independently tune different functions (RAM, minInstances, etc.). Logging is easy to search through with independent functions. Mapping URLs to functions is straight-forward via Firebase Hosting. etc.

It seems the UX of cloud functions, deployment aside, pushes you toward separate functions being more beneficial than one big function for everything. For example, there's no APIs provided by Firebase for creating a routing layer for internally dispatching within a single function (which is somewhat non-trivial), that I'm aware of?

We're not opposed to doing that if needed, developing the relevant logging affordances, routing layer, etc, but just wondered if this was a pain being felt by others and therefore a top-of-mind concern or not. Seems maybe not?

To be clear: not trying to complain at all, just looking for some clarity/direction so we can better align our efforts with the direction of the platform.

Thanks for responding!

[Kato Richardson]

There are some use cases where users have 100+ functions, but that's not standard for an app. Most solve this by only uploading changed functions using --only as described.

More common is to have a URL rewrite strategy and a handful of functions. But it's not easy to say what a decent approach would be without understanding why the app needs 150 unique activities that are all handled at the service layer.

☼, Kato

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/7943367a-c1d4-4c04-9a26-3af944adae3cn%40googlegroups.com.

[Daniel Lee]

Hi Parker,

I want to see if "codebases" (doc) fits your setup. The idea with codebases is to partition your 150+ functions into isolated groups (= codebases) which then gives you ability to re-deploy a codebase using firebase deploy --only functions:<codebase> without having CLI bothering you with prompts like "Your source code doesn't include these functions... do you want to delete them"

I imagine that the adopting codebase to your setup to feel similar to monolith -> micorsevices migration in that you'd need a clear picture of boundaries between your functions, but hopefully with a result that makes managing your function a bit easier. We are building couple more quality-of-life improvement features on top of codebase - like a feature to skip deployment of a codebase in your project if we detect that source code haven't change - so this would be a good time to explore codebases.

You are right that we don't have built-in routing constructs - we rely heavily on existing frameworks like react so you can build your own (doc) (not very useful if you are interested in event-triggered function though).

Would love to hear back on what you think.

Daniel

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CADypTEYCyJ-%3DGXiV4%2Bt4oukcZOAiWzN-_Z6-5cYDEYkSqXDrvQ%40mail.gmail.com.

[Parker Wightman]

We'll figure it out, but it's helpful to know we're outside the norm.

In case some stats/context is helpful:

• We have 40+ cron jobs and document change hooks, and ~100+ are HTTPS functions.
• Use cases span the range of:
  • Auth layer
  • Integration endpoints for external services
  • Webhooks
  • Payment processing
  • Invoice/billing querying
  • Affiliate functionality
  • Sharing/unsharing of many resources
  • SAML
  • Enterprise-specific functionality
  • Referral systems
  • Notifying customers about various things
  • Propagating changes to documents when other documents change

2 - 15 functions for any one of those given categories and things add up pretty quick.

Some of this could be (and often is) done in-client when there's not a security concern, but having web + mobile clients means consolidating some functionality into functions that need to be shared across platforms.

I don't feel like it's that strange for a large app to need so much backend functionality as part of delivering the what's expected of a modern SaaS product. See GitHub's list of just their mutations, for example: https://docs.github.com/en/graphql/reference/mutations.

Thanks for your feedback!

- Parker

[Thomas Bouldin]

Piling on a few more thoughts to help you move forward:

1. For better or worse, Firebase already uses an undocumented API to minimize redundant uploads or docker image builds in GCF 1st gen. For v2 the quotas are better, but this optimization has not yet been implemented.
2. I heavily recommend multiple codebases because it's easy to deploy groups of functions. In addition, we will eventually skip deploying functions if we can detect they haven't changed since last "firebase deploy" and codebases will dramatically increase the number of functions that can be skipped per deploy. To enable this on an existing monolithic codebase, you can experiment with yarn workspaces. This will let you split your codebase into many private packages that spit out fewer public packages. Each of those generated packages can be a codebase. If you change a really core library that everything depends on, it will show file changes in all generated packages and everything will be deployed; if you have many libraries and you edit an ancillary one, then only a few functions will be deployed.
3. For your (not "callable") HTTPS functions, I recommend you look at the Express documentation. Our interface takes a "(request, response) => void", but Express' Router and Application types implement this as well. This means you can literally take the hundred HTTPS functions and pass their callback to a Router and make the Router a single function. That will reduce your deployment size by 2/3
4. If you don't want to interact with Routers directly, we have (softly) announced and will soon release some experimental support for Next.js. With Next.js you can have one file per API. If you have a file "/api/foobar_callback.ts" then you will get a dynamic callback at "mysite.com/api/foobar_callback". This may be another way to consolidate your 100 functions into a single "web application". We automatically stitch all those endpoints into one v2 Cloud Function (V2 functions get additional CPU and serve 80x more traffic per instance by default!)

For points #3 & 4, take that all with a grain of salt and group your HTTPS handlers with intention. Microservice architectures are good for some things; they provide much better isolation for each endpoint. If you group your functions together you will lose that isolation (e.g. a crash in one HTTPS endpoint will bring down the instance serving all HTTPS endpoints), but your deploys will be much faster and you'll have many fewer cold starts because any endpoint can be served once an instance has been loaded

[Parker Wightman]

This is helpful! We were not aware of codebases, and I hadn't seen the example of using Express, that's a great starting place. We will take a look at both.

Thanks! So nice getting feedback from a platform team, we really appreciate it.

Parker

[Parker Wightman]

Just wanted to update that we've moved forward with consolidating our API into fewer endpoints (one, for now, and we might explore codebases later) and all your responses were very helpful in knowing how to move forward. We're using Express for now to handle the routing. I have questions around logging, but I'll start a new thread for that.

Thanks again!

[JP Ventura]

TL; DR

1. Don't use Express.JS (being there, done that).
2. It is a really strange scenario to deploy all functions at once (an emergency Snyk I/O or SonarQube patch, for example).
3. If you are using Firebase Functions as a RESTful API, there are great chances you are doing the work already done by Firebase Realtime Datastore or Cloud Firestore.

DESCRIPTION

Your question is so cool/strange that it really looks like a fintech product I had to work on (and I promise I shall give the proper attention ASAP 😉).

The problems I see with a lot of companies that start using Firebase is becoming hostage of the platform due bad architecture, not due the product itself. Just like Firebase or Parse), Google Firebase is a reasonable (and really cheap) trade-off between DevOps and Backend engineering, but you will hit the roof at some point (due cloud cost, engineering cost or both).

The image above it's from a 2017 presentation, but it probably will give you an insight: you can also take advantage of Google Cloud (from Appengine to Kubernetes) in baby steps, especially considering free traffic inside Google Cloud infrastructure. 

I will try to translate and update this short course at maybe it be useful to you too 👊

--

You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/a72a9525-f763-4a98-90cb-a357e607d25en%40googlegroups.com.

--

JP Ventura
Software Architect - University of Campinas

https://br.linkedin.com/in/jpventura

Computer games don't affect kids.
I mean if Pac-Man affected us as kids, we'd all
be running around darkened rooms, munching magic
pills and listening to repetitive electronic music.

Kristian Wilson, Nintendo, Inc. 1989.

=============================================

WARNING
This message is exclusively destined for the people to whom it is directed, and it can bear private and/or legally exceptional information.

If you are not addressed to this message, you are advised to not release, copy, distribute, check or, otherwise, use the information contained in this message. Sharing it without parties permission is not only illegal, but also CCPA, GDPR, and LGPD violations.

If you received this message by mistake, we ask you to return this email, making possible, as soon as possible, the elimination of its contents of your database, registrations or controls system.

The message that bears any mandatory links, issued by someone who has no representation powers, shall be null or void.

[Parker Wightman]

Hi,

Can you be more specific about what problems you faced when using Express within Firebase Functions?

> Your question is so cool/strange that it really looks like a fintech product I had to work on (and I promise I shall give the proper attention ASAP 😉).

Not fintech, a tech SaaS product.

> If you are using Firebase Functions as a RESTful API, there are great chances you are doing the work already done by Firebase Realtime Datastore or Cloud Firestore.

We aren't using them as a RESTful API, our use cases for cloud functions fall pretty strictly into these two categories:

1. Security sensitive operations that should not be done client-side
2. Complex operations that need to be shared across multiple clients (web/mobile/etc)

We utilize Firestore very heavily and (I think) are making sound engineering trade-offs as to what we do in functions and what we do in the client.

At some point we will certainly outgrow Firebase Functions for some/most of this stuff, no doubt, but I don't think we're quite there yet. More specifics about problems you faced would be helpful though!

Thanks,

Parker