Proposal: Hook for 'pre application list start'

[Isak Sky]

It would be great if there was an easy way to execute logic before any application in the mix application list is started.

I know about :start_phases in mix.exs/application, but this requires that you take move applications from :applications to :included_applications, then start them all yourself.

I have also heard there are ways to do it for releases, but it would be great to be able to do it in development as well the same way.

If this feature was added, it would enable better ways to do configuration. In the current situation, you are at the mercy of other developers adding support for {:system} type tuples, if you want to use that approach, for example. Some developers don't support it at all, some support ones with 2 elements, and not with 3, or support it for some configuration properties, and not for others (hi Ecto). Overall, it seems like it is a little harder to reason about than it needs to be.

This feature would enable a convenient escape.

[José Valim]

This is an interesting idea, we would also need to sync this with releases. We are also trying another approach with Phoenix 1.3, which is to customize how Ecto or Phoenix works on your application start, before starting the repo or endpoint in your app. The advantage of doing so in your code directly is that it makes it clear that it is a runtime configuration. However, it would allow you to customize only what you explicitly start.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/20ea6bec-0e1e-44a3-881a-1561768fd6c2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[James Fish]

Hi Isak.

You should be able to use start phases without moving :applications to :included_applications. Can you explain why moving an application to use start phases without making other changes does not work?

James

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4%2ByZ%2BoyKaoBLcr1R7FY8OWKF6ZV2pgARD3U%2Bj8uJpuheQ%40mail.gmail.com.

[James Fish]

Ah sorry read too fast, you want to be able to run the phases before starting dependencies.

[Isak Sky]

I found some related writing about this issue in Elixir. This is from the docs of conform: 

https://hexdocs.pm/conform/getting-started.html#files-files-everywhere

In the beginning, there was sys.config, and it was good. You could easily configure your application in one file, and tell the VM to use that file when booting up. It lacked runtime dynamism though (i.e. environment variables), for which you needed to use other VM flags to provide such configuration values. Even then, one still couldn’t inject configuration via the environment as any type other than a string. To this day, sys.config is still the primary means of configuring the Erlang VM.

 

Along came Elixir, a fresh take on the venerable Erlang. Along with it, came config.exs, as the solution to all the ills of sys.config. Or so it seemed. It solved runtime dynamism, and extracting typed configuration from the environment, by allowing you to execute arbitrary Elixir code in the config file when it was evaluated. A miracle! However it has one flaw, and it unfortunately undermines a lot of the power that config.exs provides: it is intrinsically tied to the Mix project structure, and thus cannot be used with releases, which require sys.config. The solution is to evaluate config.exs and convert the resulting configuration to sys.config - but now you’ve removed the ability to do any runtime configuration by fetching variables from the environment. Ultimately this leaves us where we started.

 

It is at this point that various workarounds developed, for example, the convention of configuring via {:system, "VAR"} tuples. Unfortunately, this convention is neither universal, nor consistent - you are ultimately at the mercy of your dependencies in this regard.

[Dave Cottlehuber]

On Mon, 6 Mar 2017, at 16:23, Isak Sky wrote:
> I found some related writing about this issue in Elixir. This is from the
> docs of conform:
>
> https://hexdocs.pm/conform/getting-started.html#files-files-everywhere

I've been pondering a similar issue:-

How to inject passwords and more particularly, tokens with a limited
lifespan, into Elixir? We are using hashicorp's vault [1] to handle
application secrets, so my current thinking is this approach:

- ansible provisions app / app upgrade and hands the app a master token,
scoped for this app, to a vault module
- module then uses this master token to look up various app-specific
secrets (which are time limited)
- module stores these in con_cache[2] and uses process_flag(sensitive,
true) (see [3]) to keep them out of debug/dump logs
- module is responsible for ensuring the master token is renewed every
1/2 life of the secret time limit
- module is responsible for ensuring the subsidiary secrets is renewed
every 1/2 life of their individual time limits
- inside the app, each subsidiary system simply calls our module which
internally retrieves the latest secret from con_cache's ets table

From an OTP dependency, I would then end up making this app one of the
first supervisors to start up, so that the remaining apps have their
secrets already available.

I also have a similar need to pull arbitrary, changing, runtime
configuration into apps as well, not just secrets, so
ideally this sort of functionality (lookups, app cache) would be built
into Elixir directly so that any module could use it agnostically,
without having to know that these secrets are not just from environment
variable, or sys.config but taken from a different module. Perhaps its
possible to use the same sort of approach as with logging - batteries
included, override as needed? I've not done this before so tips are
welcome.

A+
Dave

[1]: http://vaultproject.io/
[2]: https://github.com/sasa1977/con_cache
[3]: http://erlang.org/doc/man/erlang.html#process_flag-3

[José Valim]

The overall design sounds correct.

It is important to highlight that not all configuration is equal though. We can break them in three categories:

* compile time - such as logger calls purging

* runtime and stateful - such as Phoenix port. If you change the port configuration, Phoenix won't magically start listening on a new port

* runtime and stateless - when Application.get_env is called whenever it is needed 

So you could either have all apps reading from your config cache or have the config system consistently read from the external service and update Application.get_env but keep in mind some changes won't be reflected unless you recompile the code or restart some services.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/1488875563.761822.902936616.17E1418F%40webmail.messagingengine.com.

For more options, visit https://groups.google.com/d/optout.

--

[Dave Cottlehuber]

On Tue, 7 Mar 2017, at 10:30, José Valim wrote:
> The overall design sounds correct.
>
> It is important to highlight that not all configuration is equal though.
> We
> can break them in three categories:
>
> * compile time - such as logger calls purging
> * runtime and stateful - such as Phoenix port. If you change the port
> configuration, Phoenix won't magically start listening on a new port
> * runtime and stateless - when Application.get_env is called whenever it
> is
> needed
>
> So you could either have all apps reading from your config cache or have
> the config system consistently read from the external service and update
> Application.get_env but keep in mind some changes won't be reflected
> unless
> you recompile the code or restart some services.

Thanks José for the clarification. In Apache CouchDB, we wired up the
configuration to the appropriate supervision trees, and exposed the
runtime config via authorised RESTish API, so (for example) changing
the bound IP address or port results in a restart of the mochiweb
supervisor tree. This is pretty cool for long-running services like
databases so you don't lose the cached file data.

A+
Dave

[José Valim]

That's very neat indeed and is a perfect example of the discussion at hand. Some configuration can only be correctly propagated by writing additional code.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

Dave

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/1488887025.802096.903105656.579C50D2%40webmail.messagingengine.com.