Plan for automating release builds, removing NodeJS for prod setups

[Pavish Kumar Ramani Gopal]

Hey team,

I've been thinking about steps to automate release builds and to remove NodeJS & npm dependency from prod installations. Here's my high level plan on how to do this:

Automation to publish release images:

• When a tag is made and pushed to GH, build and push images (tagged as latest) to DockerHub.
• Use a mathesar_bot Docker account to do this.
  
• GH action:
• Trigger: On new tag
• Perform: Clone repo, check out tag, log into DockerHub, build and publish the images.
• For making the release itself on GH, I'd personally prefer it to come from a human.
• GH shows who made each release and it'll be good for it to come from different maintainers instead of a bot account.

Removing NodeJS from Production docker image

• I've already raised a PR to do this: https://github.com/mathesar-foundation/mathesar/pull/3474
• This has a >28% reduction in final image size.
• The frontend files are still built during the Docker image creation instead of downloading it from a hosted location to keep the image building process atomic and independent.

Removing NodeJS from 'Install from scratch' installation method

• When a release is made on GH:
• We'll have a GH action to build and add frontend files to the release assets.
• In the Install from scratch docs:
• Add a step to download frontend static files into the correct location.
• Remove steps that install NodeJS and npm.

Bonus: Make Caddy optional in 'Install from scratch' method

• Caddy is used here mainly to serve the static files.
• It would be better to serve them from gunicorn, make Caddy optional, and let the user decide their prod setup.
• This will vastly reduce the complexity in this installation method.

Let me know what you all think.

I've not yet made any basecamp tasks since we've not had a team wide discussion on this and on who's going to be doing them. However, I'd like to start work on these.

Unless I hear any objections or suggestions, I'm going to be making progress around these items this week.

[Sean Colsen]

Here are some thoughts I have…

• This thread seems to be working towards two (potentially independent) goals:
  • (A) Moving the front end build step from installation time to release time
  • (B) Increasing automation for our release process
• I support working on (A) right away, and your plans there sound great.
• I definitely support moving in the direction of (B), but I have quite a bit more thoughts about this, which would be easier to discuss on a call.
• Along with these changes to our release process, I’d like to suggest that we also consider this additional change in conjunction:
  • (C) Make the release process atomic
• My inclination would be to figure out (A) and (C) for Mathesar 0.1.6 without attempting to address (B) immediately. Then, once we have a straightforward release process that is fully documented with copy-paste commands, we work on (B) for Mathesar 0.1.7.
  

​

[Kriti Godey]

This all sounds good to me, thanks for coming up with this plan Pavish. Please send the final plan here after you, Brent, and Sean discuss tomorrow.

[Brent Moran]

A minor concern I have about this: Given that we want to publish the static files as an asset, wouldn't it be more consistent to use that asset as part of the "release" docker image creation rather than a multistage build? That would simplify the docker build, and also provide a more credible guarantee that the static files served by the docker image are the same as those published in the release. I guess I'm not convinced that the independence of the Docker build from the release assets is a good idea.

It may not be feasible to get using the asset in the docker build organized for this release, though, because there's kind of a circular dependency involved if we want to publish the docker image before actually cutting the release, or test the docker build process. We'd need to reorder those steps for it to work.

Brent

[Pavish Kumar Ramani Gopal]

> That would simplify the docker build

I disagree. It would make the docker build dependent on the static files, introduce circular dependency (as you mentioned), and will complicate the build process. Reordering our process is not a good idea since that'll involve making the GH release over untested code. I think keeping them independent is the simplest and most efficient way.

> wouldn't it be more consistent to use that asset as part of the "release"

I agree that it'll be consistent, but I do not see why that's important to ensure.

>. I'm not convinced that the independence of the Docker build from the release assets is a good idea

If we are to include a step to download static files, they cannot be part of the GH release assets. They'll have to be hosted elsewhere and that infra needs to be maintained regularly, which adds an additional organizational overhead.

[Pavish Kumar Ramani Gopal]

Brent, Sean, and I had a call on this today. Here's what we arrived at.

Docker installation: We all agreed on the multi-stage build setup.

Install from scratch:

• We'll make a draft tag and release on GH when cutting a branch.
• This draft release will include the built frontend assets. (This will be useful for QA)
• When publishing the release, we will rename the draft to the release version.
• If new code is pushed to the release branch after QA, we will delete the draft tag and create another one.
• For this release, I will try to automate this process as much as possible.
• I will also update the release process docs.

[Kriti Godey]

Sounds good, thanks for the update.