Considering SECRET_KEY and one-click deployments

[Zack Krida]

Hi team,

As we work on adding one-click, zero-config deployment options to Mathesar, we need to address how we manage the SECRET_KEY environment variable. We do not want users to have to set ENV vars manually. That level of technical knowledge defeats the purpose of one-click installation entirely.

There are a couple of directions I can see for how we can avoid users needing to set this manually:

1. Set a default (unsafe, bad) key
   We could define a default SECRET_KEY in the various configurations for one-click environments. This would allow the deployment to succeed out of the box. Then, we could warn users post-install that they should “harden” their installation by replacing the key and restarting Mathesar on whatever platform they're using. This gets Mathesar working right away without making people mess with environment variables, but it at best defers the problem and at worst creates a bunch of insecure Mathesar instances in the wild. 

2. Auto-Generate and Persist a Key
   We could auto-generate a secure key at first launch and persist it somewhere: either to the filesystem (e.g., a config file in the container or mounted volume) or to the database (seems weird, idk). This would essentially be a "more secure" default as opposed to the first solution, and the user could still override it by setting the ENV var on their own later on.

I'm curious what people think about this problem and if you see any other solutions. Or, if you see any flaws with the solutions I've proposed that I've missed.

Thanks,
Zack

[Pavish Kumar Ramani Gopal]

 I do not think we should set a default or persist on the first launch, I think we should auto-generate during deployment time.

In our Installation script (from the uv PR), we generate an `.env` file during installation. The product upon starting, takes in environment variables from both the file and the system's environment variables.

I think most 1-click deployment options would have a configuration for similar behaviour, while deploying.

For eg., DigitalOcean one's click deployment has a `build_command`in their deploy.template.yaml, where we'd probably build the containers from the docker-compose file (Note: This is separate from the `run_command` which starts the containers). I think we should provide a small shell script to `build_command` that auto-generates the SECRET_KEY, creates a `.env` file and builds containers from the docker-compose file. That way we wouldn't have to provide any default values for SECRET_KEY and make no changes to Mathesar itself.

[Brent Moran]

I think (2) is the only acceptable solution of those options. The persistence should be either a config of the platform (if relevant) or a file that sets environment variables. I think that if we can't get one of those working for a given platform (aas) for the moment, we should skip it.

As for other solutions, we might be able to (as Pavish suggested) set up a secret key generation on a platform-by-platform basis. I don't like that solution since it means a different way to configure the secret key (on our end) for each platform, which sucks. So, I'd rather it be something that Mathesar itself does, and persists somewhere consistent (at least from the service's perspective). I'll try to throw a quick example of what I'm thinking together before I fly out.

Brent Moran

Technical Strategy Lead, Mathesar

[Zack Krida]

Thanks for the input, Pavish and Brent. I want to talk through DigitalOcean App Platform (in-progress PR here) as an example of a further problem that's relevant to many of these deployment systems:

DigitalOcean's app platform does not have any form of persistent filesystem. This means any solution using a "build_command" or that writes the value to a file would produce a different value on every build. Any saved file with a SECRET_KEY value would go unused.

In short, the only way to set the SECRET_KEY in a persisted way is to set it as an ENV var in Digital Ocean's UI, or in the initial config definition of the service. 

Our options are then:

1. We don't set a default value for SECRET_KEY, and the user has to add one in the DigitalOcean UI either before deploying or right after deploying. Mathesar won't work until they make this intervention.
2. We set an unsafe, public value for SECRET_KEY ("update-this-later-do-not-use" or whatever") in the initial config. Users can still set this up front, but if they do not the Mathesar installation still works immediately, and they can set a secure value later on.
3. We set up Django in Mathesar to set a default SECRET_KEY if one isn't present (the same way "django-admin startproject" works) . This value will be generated without being persisted anywhere (which wouldn't work anyway), so it'll be reset every time.

The outcomes of each of these:

1. The install doesn't work until the user sets SECRET_KEY in the Django UI. This doesn't exactly feel "one-click".
2. The install works immediately. The install is insecure until the user manually sets a new SECRET_KEY value, but all sessions and cryptographic signings are persisted until they do so.
3. The install works immediately. The install is secure even before the user manually sets a new SECRET_KEY value, but all sessions are reset any time the app redeploys for any reason (container error, update, etc.)

For DigitalOcean specifically, I prefer #3 followed by #2. For the DigitalOcean one-click setup to be suitable for production, there's likely actions the user will want to take in the Digital Ocean UI anyway that feel comparable in complexity to setting an ENV var:

• Converting Mathesar's internal DB from a DigitalOcean "Dev DB" into a "Managed DB". This costs a few USD more, but allows for automatic database backups in Digital Ocean without having to pd_dump. 
• Configuring a custom domain name. All they need to do for this is add the domain name to the DigitalOcean UI, configure DNS, and the service will automatically redeploy. The ALLOWED_HOSTS env var is set dynamically to the domain.

@Brent Moran: is there any reason we wouldn't want Mathesar to set SECRET_KEY by default when it isn't present? If so, how could we make this work for ephemeral filesystem platforms like DigitalOcean? I could imagine something like an AUTO_SET_SECRET_KEY=true env var or some other switch to set the default.

Thanks for reading.

--

Zack Krida

Product & Community Lead | Mathesar

za...@mathesar.org | Schedule a call

[Brent Moran]

Django uses the SECRET_KEY for an encryption key for, e.g., the user password column. Changing the SECRET_KEY will lose access to those columns. Maybe there's a config or trick we can use to make it so that's not the case, but #3 is right out if we expect them to use Mathesar for long, unless we change that. Even if we change that, we'd need to persist whatever encryption key ends up being used so I'm not sure it gains us anything.

Regarding setting a secret key automatically if one isn't present: I have no problem with that, and I set up a (very quick, maybe broken) PR to demonstrate the principle. The logic is:

• If  there's a secret key in the environment, we use that,
• Else if there's a secret key persisted in config/settings/secrets/secret_key.py, we use that,
• Else we generate a new secret key and persist it in that file (from the Mathesar web service's perspective; will not work if the directory/volume are ephemeral).

The example PR uses a docker volume for the config/settings/secrets/ directory.

Questions: 

• Is there no such thing as a persistent volume for the DigitalOcean one-click setup? That seems ridiculous to me. 
• A brief glance at their docs has some blather about persistent block storage for k8s setups. Could we try that?
• Is the deployment process able to run arbitrary code? I.e., can we just generate it when they click the button and then inject it into the environment in a persistent way?

If it's just not possible to persist the secret key anywhere that the container can write, then the only way forward seems to be having a bespoke-to-DigitalOcean way to configure an environment variable and set it during the deployment, i.e., the third bullet point. If that's not possible, it's not really one-click at all. Maybe we need to reframe this as "deploy to DigitalOcean infra as easily as possible", and come at it from that angle. Give ourselves more flexibility.

Brent Moran

Technical Strategy Lead, Mathesar

[Zack Krida]

I'll write a longer reply tomorrow but the secret key isn't actually used for hashing passwords. It's only used for:

- Generating session cookies

- Creating CSRF tokens

- Signing password reset tokens

Zack Krida

Product & Community Lead | Mathesar

za...@mathesar.org | Schedule a call

[Zack Krida]

Actually, I have a minute now. So, the consequences of resetting the SECRET_KEY are that every user is logged out and any password reset links in the wild will no longer work. So not dire consequences, but still consequences inappropriate for a running production application.

That is to say that I still think the 3rd solution is likely the best. What do you think Brent, given the passwords aren't hashed with the SECRET_KEY?

If that's not possible, it's not really one-click at all 

 

I agree with this in spirit, but it's ultimately a rather gray area. Like I mentioned for real production setups there are always going to be additional steps beyond one click. My main heuristic for "one click" is that a user can press a button or two without any decision making or technical knowledge and get a running Mathesar instance. 

If they have to take a few steps to make it "production ready"--again, things they're likely to do anyway, like configure a custom domain, setup backups, and so on--I think setting an ENV var by pasting a generated string into an input, isn't too much to ask.

[Brent Moran]

It seems you're right about the user login password hashing, I was misremembering.

However, as you can see in the config/settings/common_settings.py file, we're using the SECRET_KEY as a SALT_KEY, which is then used by the django-fernet-encrypted-fields library to salt values before encrypting them. We use these fields for storing role passwords (e.g., for Mathesar to connect to the user DB).

 I checked by changing the secret key on my local dev environment, and that does in fact break things. I can no longer connect to databases using configured roles.

We could try to change that logic to detach the salt key and secret key, but I think that puts us back in the same situation (we'd just need to store the SALT_KEY instead; maybe I'm not being creative enough). Furthermore, hashing doesn't work in this case because we need to actually retrieve the DB password to send to the database. I think we might be able to submit pre-hashed passwords for DB authentication in some cases, but that depends on their database configuration. Not sure how common accepting hashed passwords is in the PostgreSQL world; we could look into that.

For context, I don't really think the pasting is the issue with customizing the secret key, it's the generating. My mental model of "this user is significantly more likely to install Mathesar if they don't have to write any config files or use the CLI" for the DigitalOcean use-case makes that irritating. We can either send them to a website for generating random strings (would you use a website to generate a password?) or give them a CLI command to run (hopefully they're not on Windows, and know how to open a terminal). Or something I'm not thinking of. I.e., to me, the whole point of the DigitalOcean concept is that you can get a usable deployment without going to the CLI.

Brent Moran

Technical Strategy Lead, Mathesar

[Zack Krida]

Ah, really good to know we're encrypting role passwords with the secret key. We definitely can't be changing it, then. I'm sure for most platforms we'll be able to set a key automatically in a few different ways:

1. Persisting to the filesystem, as in your experimental PR (which looks good, by the way!)
2. Using the platform's own API/SDK to set the ENV var (A few platforms give you authenticated access to their APIs during build, DigitalOcean does not)
3. Have the user manually set the ENV var, when necessary.

I added some steps to my Digital Ocean instructions that should streamline the SECRET_KEY flow without adding too much friction for users. There's some simple client-side JS to generate a key and copy it to the user's clipboard. Here's some screenshot examples from my PR:

It's using the browser's native crypto (as in "cryptography" not "cryptocurrency") library to generate random values. I think this is sufficiently safe as long as we instruct users not to share the keys with anyone. 

What do y'all think?