Requesting details and explanations for rails-deploy-script for the plan to design containers

[Mattias Axell]

Hi alaveteli-devs! :)

We are looking to put every single part of our Alaveteli installation in containers with e.g. Podman 

If we are to use volunteer resources for this we need to learn what the rails-post-deploy script is doing exactly and for which part.

Can we get an explanation for every part in the script such as comments in the script code explaining what the rails-post-deploy does in detail?

Some parts of the scripts are used during deploy. How can we run the deploys parts of this script when building the container image?

Because it seems that many of the scripts need database connection. And we want to run those parts that need database connection on container startup and not in the build process.
And preferably, we want to run as many "deploy" parts as possible during the build of the container image so that we can stop and start the container fast.

https://github.com/mysociety/alaveteli/blob/develop/script/rails-post-deploy

This script refers to these two:

https://github.com/mysociety/alaveteli/blob/develop/script/rails-deploy-before-down

https://github.com/mysociety/alaveteli/blob/develop/script/rails-deploy-while-down

All the best,

Mattias

[Laurent Savaëte]

Hey Mattias,

Looking at the scripts you mention below, my high-level understanding is that they setup the rails environment (paths, the name of the db to use...), installs ruby gems, compiles js and css to versioned files (the ones with a hash in the url that allow infinite caching), the second one migrates the db schema ot the new format when there are changes.

I would love to know about the rationale to encapsulate everything in containers.

My reasoning for our site has been roughly the following:

We (madada.fr) use ansible to deploy alaveteli, and almost everything runs on the same single machine. We have a couple of extra services in containers (eg. metabase, which is available as a container image or JAR) but the standard alaveteli components are not.

I have thought about moving to containers and eventually stopped, after containerizing a variety of other services, which generally made me sad :)

Containers add a layer of complexity, you need to build each image, then get them to work together ("orchestrate them") via docker compose, swarm or kubernetes or something else.

They also increase hardware requirements because of CPU/RAM overhead linked to the container management process.

Ultimately, containers are paving the way towards high availability, no downtime deployments, "infinite" scalability, etc...: you rent a bunch of servers, add them all to a cluster, throw all your containers in there, and the container orchestrator replaces the dead ones, upgrades them to the new version without users noticing, etc...

The thing is, our site does not need all this: our site is small, we will "never" reach a size where we cannot find a server powerful enough to run the entire service, and if we run into that problem, I will just move the database to a separate machine, and be fine for a while. We also have a natural maintenance window every night and weekend, where traffic is very low (usually only crawlers between 2-6am), so we can afford to stop the server for a few minutes without anyone complaining. Over the past 3 months, we've had 99.96% availability on madada.fr. None of the container based services I run are doing as well (but I also don't pay for a hosted container orchestrator like k8s which costs a small fortune).

In practice, I do our regular deployments during the day, nobody notices. I upgrade alaveteli versions on weekends, just in case something breaks.

My biggest problem for a while was the development environment which never exactly matches production. Containers are supposed to help with that, but in my experience, they just make everything tedious and replace one type of problem with another. My dev env is in a virtual machine now (yes, old fashioned!).

My current annoyance is how long the deployment takes (10-15minutes), but since it happens in the background, it's very minor. And the fact that ansible can break halfway through an upgrade, but that usually gets fixed on the staging server, so it doesn't affect production.

In summary, I decided against containers because they imply a lot of extra work that produces only limited benefits for our use case.

If I were starting from scratch now, I would probably use nix instead of ansible.

Laurent

--
You received this message because you are subscribed to the Google Groups "Alaveteli Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alaveteli-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/alaveteli-dev/CAEJghOq9cjfRE%3DXXW%2Bf7_8wgKUMS74ZprTFxoREgotyGg45mUg%40mail.gmail.com.

[Oliver Lineham]

We run fully in containers and have done for ten years.

I'm re-containerising right now to modernise the approach and did already unpick the convoluted install scripts. Very little of what the scripts do is necessary or even useful, in my view.

Alaveteli seems to be written to run everything on one box (postfix, rails, cron) which is not ideal for numerous reasons. But it's solvable. 

Instead of postfix calling rails (tightly coupled and inefficient), postfix writes to S3, and sidekiq reads from there. This makes a nice queue.

The main thing that is not very container friendly is the cron jobs (50 or so!). So I have converted every one of them to run as Sidekiq jobs instead, using sidekiq-cron.

This is also dramatically more efficient. Instead of rake under cron loading the entire rails application into memory (10 seconds) every time, they now run in a few milliseconds with no memory impact. 

Oliver

--

[Graeme Porteous]

Just to say we would welcome any contributions to help separate the
different Alaveteli components to make it easier to deploy.

At present it is designed to run everything on one box but we've
recently started deploying WDTK on two servers, we're slowly trying to
moving everything to run in background jobs to spread to load across
the servers (via Sidekiq at present, although maybe we'll move to
Solid Queue [1] in the future to follow the Rails approach and remove
the dependency on Redis).

Adding support for ActionMailbox [2] would help separate Postfix from
Alaveteli and provide better email ingress options [3] and allow users
to use 3rd party services to receive emails so people won't need to
run their own mail servers.

[1] https://github.com/rails/solid_queue/
[2] https://github.com/mysociety/alaveteli/pull/8049
[3] https://edgeguides.rubyonrails.org/action_mailbox_basics.html#ingress-configuration

--
Graeme Porteous
gra...@mysociety.org

> To view this discussion on the web visit https://groups.google.com/d/msgid/alaveteli-dev/CAOPSvnA2D9zC4r8ZiM8%2BjXuqQ3dPTUN%2B8gbwCE_LNjJz33%2B_Jg%40mail.gmail.com.

[Linda]

Hi. I tried building an alaveteli image using the "Manual installation" instructions (https://alaveteli.org/docs/installing/manual_install/). I saw the docker folder in github (https://github.com/mysociety/alaveteli/blob/develop/docker/Dockerfile), but that doesn't run rootless.

If some of you are already running alaveteli components in containers, is that available as open source? Obviously, having an official base alaveteli image would be great.

The reason we want to use containers is to track versions with image tags and for security. I.e.:

- to track the alaveteli base versions, themes, and various dependencies as a unit

- to run with readonly filesystem

- to run rootless and as non-root inside the containers

- to not expose postgresql ports (we can run postgres on the same machine, in the same network namespace as alaveteli)

- use "secrets" for passwords and keys

- to deploy as restartable services (I prefer systemd user services that run podman container processes with log-driver=k8s-file which fluent-bit can parse and forward)

Not everything have to run as an isolated container. Preferrably, all the "applications" do, but maybe not system/daemon services.

Work-in-progress is here: https://gitlab.com/handlingar/handlingar-devops/-/tree/main. It's a mono-repo for alaveteli-components (alaveteli, postgresql, nginx), to build each as an image with gitlab pipelines.

Some issues right now are:

- I can't get alaveteli to log to stdout. If i use option --log production.log it tries to create a log file "production.3000.log". Preparing a symlink "production.3000.log" to /dev/stdout still gets permission errors at runtime.

- I tried running cron inside the container (rootless). Still trying. But maybe I don't have to if the crontab can be replaced?

- It seems like every exec-rake task needs postgresql running (because of some hierarchical dependency in the tasks?), so it put those commands at startup -- which is slow.

- Installing the theme must also be done at startup and it uses git clone, and I want to avoid network calls during startup (though somewhat solved by cloning from a local directory instead)

- Trying to understand email setup