Proposal: Closing the gap between configurations

[José Valim]

Over the last releases we have been making improvements to the configuration system. However, there is still one large gap to be addressed. Today config files are treated as compile-time configuration by default. This behaviour, alongside overuse from libraries, made using the application environment via config files more rigid and error prone than they have to be.

Today, Mix will load "config/config.exs" before any of the dependencies are compiled and on every command. This makes basic scenarios like the ones below impossible:

• If your project requires some environment variables in development, you cannot enforce them. Otherwise, basic commands such as mix help or editor integration will stop working

• If you need to configure one application based on another application, that is impossible to do as well, as no applications are available at this point

Also, because configuration files run during compilation by default, library authors often accidentally rely on compile-time configuration. Luckily, Elixir v1.10 has added Application.compile_env/2. In the future we will deprecate Application.get_env/3 in the module body, which will help steer people to the correct direction in their own applications.

On the opposite side, we have releases, where the configuration works at runtime only. Given there is no way to execute configuration at runtime in Mix, we end-up with two completely disjoint approaches. This introduces a couple issues of their own:

• Since "config/releases.exs" run only during releases, if you have a syntax error (or similar), you will find about it just way too late

• There is no way for frameworks like Phoenix to provide a configuration file that works for both Mix and release deployments

Our goal is to address these issues. However, it is important to consider that a complex project today may already have many configuration files:

• config/config.exs (compile time) - shared configuration settings
• config/{dev,test,prod}.exs (compile time) - per env configuration settings
• config/releases.exs (runtime) - release specific configuration settings

Therefore, we would like to propose a new configuration file that can address the problem above while replacing the need to use "config/releases.exs" in 99% of the cases.

Proposal: introduce config/runtime.exs

Our proposal is simple: we will introduce "config/runtime.exs". "config/runtime.exs" will be loaded both by Mix and Releases, closing the gap between them.

For Mix, "config/runtime.exs" will load after the code is compiled and before the application starts, this allows "config/runtime.exs" to rely on code from dependencies, as long as you keep in mind that any application that is started during "config/runtime.exs" cannot be configured by "config/runtime.exs" itself. Furthermore, given "config/runtime.exs" works at runtime, changing it won't require the whole application to be recompiled.

For Releases, it will work precisely the same as "config/releases.exs". If both are available, "config/runtime.exs" is executed first, followed by "config/releases.exs".

There are a couple pitfalls to be aware though:

• Since "config/runtime.exs" is used by both Mix and releases, it cannot configure :kernel, :stdlib, :elixir, and :mix themselves. Attempting to configure those will emit an error. For those rare scenarios, you will need to use "config/releases.exs" - but "config/releases.exs" will remain simple, which will reduce the odds of syntax errors.

• Since "config/runtime.exs" is used by both Mix and releases, it cannot invoke "Mix" directly. Therefore, for conditional environment compilation, we will add a env/2 macro to Config that will be available for all config files. For example, instead of a "config/runtime.prod.exs", one will have to:

  import Config
  
  env :prod do
    config :my_app, :secret_key, System.get_env!("SECRET_KEY")
  end

One may argue that "config/runtime.exs" should eventually replace "config/config.exs" as the default file for application configuration. We will certainly evaluate this option in the future but it is important to take baby steps. And the first step is to support "config/runtime.exs". :)

Implementation considerations

This section covers implementation details. It is not part of the proposal per se. Although the feature is relatively small, it requires many improvements to Mix and the underlying config engine. The tasks are:

• Load config/runtime.exs inside Mix
• Copy config/runtime.exs inside escripts and load them when the escript runs
• Copy config/runtime.exs inside releases (similar to config/releases.exs)
• Add a feature to Config.Reader that allows a warning to be emitted if an undesired module is used (for example, Mix)
• Add the env/2 macro to Config
• Raise if "import_config" is used in "config/runtime.exs" and "config/releases.exs" - providing proper guidance to users

One aspect to consider is exactly when runtime config should be loaded inside Mix. We need to choose between doing it after the "compile" task or before "app.start". The issue is that many projects have tasks that only need the application to be compiled but not started. For example, Ecto Repo management tasks or Phoenix routes tasks. Those tasks today simply run Mix.Task.run("compile"). However, if we were to introduce "config/runtime.exs" and load it before "app.start", those tasks will now run without "config/runtime.exs" and behave incorrectly.

Therefore there is an argument to be made to load the runtime configuration right after the code is compiled - even though this is a bit unintuitive. The other option is to ask users to always run "app.start" as the entry point and pass the "--no-start" if they actually don't want to start their apps, which is also a bit counter intuitive. Unfortunately, the second option means projects will behave incorrectly until they are updated.

[Paul Schoenfelder]

This was always the hardest/most annoying part about the release tooling, and I think this is probably an improvement, but it does seem to me to trade one set of annoyances for another. That said, I think I'd prefer the new set.

My take is that the config should be evaluated after compilation, after applications are loaded, but before any applications are started, as this can be handled properly both when run under Mix and when run in releases by having the boot script invoke the configuration entry point.

For those cases where one needs an application started in order to configure the system (which I would assert indicates a different problem), then one can use the `Application` APIs to temporarily start the application, perform that config, and then stop the Application so that the rest of startup can proceed normally (this is more important for releases than under Mix, but I think its important to treat them the same).

It's also my experience that most of the time you only need the applications _loaded_, not started, since the APIs you want to call during the configuration stage typically don't require the system to be running (and wouldn't ever, if I had my way). That isn't universally true today of course, but a proper solution to that  particular problem is going to require some concrete examples that aren't themselves resolvable by other, more appropriate, means.

Anyway, that's my two cents :)

Paul

--

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4%2BS7nefgLCXCb9OWQGj9d9eE1TrDstq%3Dkm%3D%2B2a7iT9DAw%40mail.gmail.com.

[José Valim]

Hi Paul!

I believe we are mostly in tune then!

> My take is that the config should be evaluated after compilation, after applications are loaded, but before any applications are started, as this can be handled properly both when run under Mix and when run in releases by having the boot script invoke the configuration entry point.

This is a good point. But if we move in this direction, then we will definitely need to introduce a "mix app.load" task, which means existing projects won't leverage runtime config unless they invoke "mix app.load" instead. Although I like the dichotomy here: users have to call either "app.load" or "app.start" in their Mix tasks. This also helps avoid subtle differences in behaviour, so I really like it.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/d92115bd-a9fc-4236-b041-a80aa5fc5610%40www.fastmail.com.

[Stefan Chrobot]

What do you think about renaming "config.exs" and friends to something like "compiletime" (or "compile", "static" "build", "mix")? I see the proposal to add "runtime.exs" as a way to embrace the fact that there are two separate concepts: build configuration and runtime configuration, so why not make it even more explicit? It would make things like reading ENV during build time more obvious (i.e. System.get_env in build.exs).

Best,

Stefan

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4JR65fkD_uyOKup1NqcNcJGywt8xe6ZDC-tdXtnvV_M4w%40mail.gmail.com.

[José Valim]

Hi Stefan,

That's a possibility, yes. If we make config/runtime.exs the main entry point, then we will want to move away from config/config.exs, and renaming it to config/compile.exs would be a good idea. But right now it is too early to commit to such a change. We need to validate config/runtime.exs first.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CACzMe7aMLEKTMXaGcENBjSFuei69V9g-VOqNNuwAqmWyo_cq2w%40mail.gmail.com.

[Dmitry Belyaev]

> Since "config/releases.exs" run only during releases, if you have a syntax error (or similar), you will find about it just way too late

We solved this problem by setting dynamic runtime configuration ONLY in "config/releases.exs" like so

config :my_app, base_url: URI.parse(System.fetch_env!("BASE_URL"))

And "dev.exs" sets some defaults and uses the same releases.exs to configure applications:

[{"BASE_URL", "http://localhost"}]

|> Enum.each(fn {key, value} -> System.get_env(key) || System.put_env(key, value) end)

import_config "releases.exs"

Of course, when used like that, it wouldn't affect the already started applications which rely on configuration during the startup (as you mention :kernel, :stdlib, :elixir and :mix).

However we don't import "releases.exs" in other environments:

* "prod" is only used for compilation on CI, so there's no need to load dynamic runtime configuration, and it contains only production-grade compile-time configuration and non-configurable runtime configuration;

* "test" contains its own set of settings completely separate from the rest but it's possible to `import "prod.exs"` if that reduces repetitions.

So I suppose our usage of "releases.exs" is similar to what you are proposing with "runtime.exs".

However I think using it unconditionally in every environment will only complicate things, as you mention we'd have to branch on Mix.env/Config.env - this seems more complex than our current approach.

> In the future we will deprecate Application.get_env/3 in the module body.

Does that mean during module compilation? I guess it'll still be available in normal runtime?

As I understand "runtime.exs" is supposed to be simply copied to the release.

Although it looks reasonably safe, I definitely wouldn't want to see some testing or development settings in a file included in a release especially if they are unused.

Maybe we should come to this problem from a different side.

What if we introduce a separate "compilation" stage (using separate set of environments - e.g. dev_compile/prod_compile) and start a clean VM for the real runtime after Mix compiled the applications?

Kind regards,

Dmitry Belyaev

--