Proposal: Moving towards discoverable config files

[José Valim]

One of the major differences between running your application as a release and as a Mix project is the differences in configuration. Mix evaluates the configuration right before the application starts, releases evaluates the configuration when your application is compiled.

This implies in a large mismatch of how those two environments are used. For releases, environment variables (read by `System.get_env/1`) need to be set when the application is compiled and such information may not be available at this point.

Ideally, we would want a release to evaluate the configurations files in `config` when the release starts. One approach would be to copy the configuration files as is to the release but that's hard to achieve in practice for two reasons:

  1. A config file may import other config files and often importing those files happen dynamically. For example: `import_config "#{Mix.env()}.exs"`. The dynamic import makes it hard for release tools to know which configuration files must be copied to a release, especially in cases like umbrella projects, where a developer may load configuration across projects

  2. Even we copy today's configuration files to a release, those configuration files rely on `Mix`, which is a build tool and therefore it is not available during releases

To solve those issues, we need to make sure we can discover all imports of a configuration file without evaluating its contents. We also need to introduce a new module for configuration that does not depend on Mix.

This is the goal of this proposal.

## Application.Config

This proposal is about introducing a module named `Application.Config`. It will work similarly to the existing `Mix.Config`, except it belongs to the `:elixir` application instead of `:mix`. This allows releases to leverage configuration without depending on Mix.

The user API of `Application.Config` is quite similar to `Mix.Config`. There is `config/2` and `config/3` to define configurations. There still is `import_config/1` to import new configuration files with one important difference: the argument to `import_config/1` must be a literal string. So interpolation, variables or any other dynamic pattern is no longer allowed.

In order to help with configuration management, we will introduce a project option in your `mix.exs`, named `:config_paths` to help manage multiple required and optional configuration files.

In the next section we will provide an example of how configuration files used by projects like Nerves and Phoenix will have to be rewritten and then we will discuss how integration with release tools such as distillery will work.

### A common example

Projects like Nerves and Phoenix generate files with built-in multi-environment configuration. Today, this configuration has an entry point `config/config.exs` file that imports an environment specific configuration at the bottom:

# config/config.exs

use Mix.Config

config :my_app, :some_shared_configuration, ...

import_config "#{Mix.env()}.exs"

And then each `config/{dev,test,prod}.exs` provides environment specific configuration. For instance:

# config/dev.exs

use Mix.Config

config :my_app, :some_dev_configuration, ...

The issue in the example above is the use of dynamic imports, such as `import_config "#{Mix.env()}.exs"`. We will address this by defining both `config/config.exs` and `config/#{Mix.env()}.exs` as configuration entry points in your `mix.exs`:

# mix.exs

def project do

  [

    ...,

    config_paths: ~w(config/config.exs config/#{Mix.env()}.exs),

    ...

  ]

end

And now we can define those configuration files without dynamic imports:

# config/config.exs

use Application.Config

config :my_app, :some_shared_configuration, ...

# config/dev.exs

use Application.Config

config :my_app, :some_dev_configuration, ...

In Phoenix, the `config/prod.exs` case may link to a separate `prod.secret.exs` file. While we could also refer to this file in the `:config_paths` configuration in the `mix.exs` file, because it is only specific to production, it is more straight-forward to continue importing it at the bottom. So a `config/prod.exs` would look like this:

# config/prod.exs

use Application.Config

config :my_app, :some_prod_configuration, ...

import_config "prod.secret.exs"

By adding `:config_paths`, we are able to move the dynamic configuration to the `mix.exs` file and make the order that configuration files are loaded clearer.

### A FarmBot example

Nerves projects tend to rely extensively on configuration files. So let's look into existing open source Nerves projects and see how this proposal will fare. Let's take a look at [FarmBot v6.4.1](https://github.com/FarmBot/farmbot_os/tree/v6.4.1).

The questions we want to answer are: if we move the FarmBot project to the proposed `Application.Config`, will they be able to express of all the existing idioms they do today? And, even further, will their configuration files become simpler or more complex?

From looking at its [config/config.exs](https://github.com/FarmBot/farmbot_os/blob/v6.4.1/config/config.exs), we can already see a pattern that won't work in releases: [the use of `Mix.env` and `Mix.Project.config`](https://github.com/FarmBot/farmbot_os/blob/v6.4.1/config/config.exs#L3-L5).

We can see [those variables are used to dynamically import configuration](https://github.com/FarmBot/farmbot_os/blob/v6.4.1/config/config.exs#L69-L77), which `Application.Config` won't allow.

Those idioms are perfectly fine with how configurations work in Mix today. But they will no longer with a release built on top of `Application.Config`.

The solution is to move all of those imports to the `:config_paths` option in `mix.exs`. However, note that some of those dynamic imports are optional, so we will also need the ability to explicitly tag them as such:

# farmbot/mix.exs

def project do

  [

    ...,

    config_path: ~w(config/config.exs config/#{Mix.env()}.exs) ++

                   optional_config_paths(@target, Mix.env()) 

    ...

  ]

end

defp optional_config_paths("host", env),

  do: [{:optional, "config/host/#{env}.exs"}]

defp optional_config_paths(target, env),

  do: [{:optional, "config/target/#{env}.exs"}, {:optional, "config/target/#{target}.exs"}]

We believe this approach is an improvement to the previous one because it allows all environment and target specific handling to remain in the `mix.exs` file and not scattered around multiple configuration files.

## Using it in releases

In the previous sections, we have outlined `Application.Config` which no longer depends on Mix and has a restricted `import_config`.

Now that we are able to see all of the configuration files that affect our system, a release tool, such as discovery, should be able to traverse all of those configuration files and merge them into a final `config/release.exs` that will be part of your release. In fact, Elixir will provide a convenient API that performs such operation, streamlining the release assembling process. 

## Unresolved topics

There are two important topics that we have not included in this proposal and they will be discussed in a further step.

  1. What about umbrella projects? Umbrella projects also rely on configuration and we need to make sure the listed mechanisms also work well with umbrellas.

  2. How to avoid common pitfalls? Even though we will migrate to `Application.Config`, there is nothing stopping a developer from accessing Mix (and the module defined in the `mix.exs` file) from their new config files. As we have seen, this may lead to errors when running releases, as releases do not have Mix available. To address this, we may introduce checks when assembling releases that make sure `Mix` is not invoked in configuration files, raising appropriate error messages in case they do.

## Summing up

We propose a new `Application.Config` module and a new `:config_paths` project option that allows release tools to discover all of the relevant configurations in a system. Release tools can then merge and copy those configuration into releases and execute them as part of the release process, allowing dynamic calls such as `System.get_env/1` to work in development and in production transparently, with or without releases.

José Valim

www.plataformatec.com.br

Founder and 

Director of R&D

[Austin Ziegler]

This is an interesting idea, but I’d like to pick up Saša Jurić’s [call](http://www.theerlangelist.com/article/rethinking_app_env) to move as much as possible from static application configurations and toward more dynamic runtime configuration. For the main Elixir app that I deploy, we use distillery to generate the release, and then generate an AMI image with Ansible scripts that customize the installed sys.config (this does periodically result in issues where the Ansible sys.config needs updating). We are looking at shifting toward Consul and Vault, but that is still some time away for us, so the continuity of sys.config is more important to us than anything else until that point, and when we do switch, I’m not sure that this will provide anything meaningful for us.

(This may be more appropriate for Heroku and Heroku-like deploy environments, but I am increasingly inclined to avoid those as much as possible.)

-a

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4L%2BAsNewcj%2BPF0fogpgDQEAc6WDMDy7X-W5evx19WPkrA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Austin Ziegler • halos...@gmail.com • aus...@halostatue.ca
http://www.halostatue.ca/ • http://twitter.com/halostatue

[Allen Madsen]

FYI, most of the discussion is happening on the forum: https://elixirforum.com/t/proposal-moving-towards-discoverable-config-files/14302/73

Allen Madsen
http://www.allenmadsen.com

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4L%2BAsNewcj%2BPF0fogpgDQEAc6WDMDy7X-W5evx19WPkrA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Austin Ziegler • halos...@gmail.com • aus...@halostatue.ca
http://www.halostatue.ca/ • http://twitter.com/halostatue

--

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAJ4ekQuQ5F0zCQyyd6ijaCj9hPryqJ7tZk3FzZN55btXKXvW%3DA%40mail.gmail.com.