How to minimize RAM usage during Go binary compilation

[Jason Kulatunga]

Hi,

I’m doing something a bit unusual with my application and I’m trying to figure out if there’s a better way to do this that I’m missing.

Basically my application Fasten Health is designed to allow patients to pull their medical records from healthcare institutions — of with there are 10,000 currently supported, and 100,000s more that we'd like to support in the US.

We have a repo/library called fasten-sources which is made up of mostly generated code (using dave/jennifer).
It has a couple of primary packages:

• pkg which contains a list 10,000 enums, one for each support healthcare institution.
• definitions/internal/source which contains a public function defining the OAuth configuration for each healthcare institution
• clients/internal/source which contains a public function effectively instantiating a custom API/Http client for each healthcare institution

All external references to this library use a Factory pattern - https://github.com/fastenhealth/fasten-sources/blob/main/clients/factory/factory.go  — so the public functions do not need to be public in reality.


At this point the build of our application (which leverages this library) is causing OOM issues during our Docker image build on Github Actions and custom Depot.dev build nodes (64GB of memory)

Here’s what I’m wondering:

1. Could this OOM issue be caused by the number of public symbols in the packages? As we start supporting more and more institutions, I’d expect this number to grow by orders of magnitude, which will definitely cause even more OOM issues
   • My functions/structs all match an interface, so if I change my public functions to private functions, would that solve the problem in a permanent way?
2. Is the OOM issue caused by the number of files in a package? Instead of having 100,000s of individual files, if I had 1 massive file, would have have any impact?
3. Is the OOM issue caused by the number of symbols in a single package? Should I find a way to distribute the enums, functions across multiple packages would have have any impact?
4. Does the `internal` path have any impact on RAM usage during compilation?
   
5. Instead of using a switch in the Factory for instantiating the clients and definition files, is there some other compiler friendly way to do this instead?

Basically I recognize that I'm doing something a bit unusual, but I cant find a way around it. I need to support 100,000's of institutions in the future, so bandaid fixes (or "just use a bigger build machine") won't work for us. A fundamental refactor of the repo is completely acceptable, since its all generated code. I just don't know which changes will actually impact the amount of RAM used during compilation

Thanks!

-Jason

[Sean Liao]

considering it is primarily static information, maybe it should be passed in and as data during run time, instead of being embedded and compiled as code?

- sean

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/f870ec97-5ac5-4211-9379-5f2c0dd97e0bn%40googlegroups.com.

[Diego Joss]

Or it could be embedded using package embed (https://pkg.go.dev/embed). Just another candidate solution.

-- Diego