Application.get_env/2 support for {:system, "VARIABLE"}

[Cory ODaniel]

I've definitely run into issues where I need to pass an environment variable at run time, but an application that I use doesn't support the {:system, "DATABASE_URL"} or {:system, "PORT"} style environment variables that Ecto and Phoenix support.

I'm curious if it would be beneficial to add support in Application.get_env/2 that when the value that returns matches {:system, var} then System.get_env(var) would be called under the hood.

[José Valim]

There has been a couple discussions on the topic either here or on the issues tracker.

The consensus is that this problem needs to be solved but we are not quite sure how. The only way to support {:system, "DATABASE_URL"} in a way that it would also work for Erlang applications is by hijacking the application controller using private APIs. We could also try solve this exclusively for Elixir but then there would be gaps where it wouldn't be supported.

Ecto 2.1 is trying a new approach where the value is configured using a repository callback, that's what we will try to do when Phoenix 1.3 comes out and see where it will lead us to.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

On Sat, Dec 17, 2016 at 1:48 AM, Cory ODaniel <co...@coryodaniel.com> wrote:

I've definitely run into issues where I need to pass an environment variable at run time, but an application that I use doesn't support the {:system, "DATABASE_URL"} or {:system, "PORT"} style environment variables that Ecto and Phoenix support.

I'm curious if it would be beneficial to add support in Application.get_env/2 that when the value that returns matches {:system, var} then System.get_env(var) would be called under the hood.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/9fd6e998-04d2-4d7d-87ee-34abb16ee779%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Louis Pop]

Why not just use `Application.get_env(app_name, var_name) || System.get_env(var_name)` in your init callback? Seems much clearer to me and requires no additions to get_env.

On Sat, 17 Dec 2016 at 09:17 José Valim <jose....@plataformatec.com.br> wrote:

There has been a couple discussions on the topic either here or on the issues tracker.

The consensus is that this problem needs to be solved but we are not quite sure how. The only way to support {:system, "DATABASE_URL"} in a way that it would also work for Erlang applications is by hijacking the application controller using private APIs. We could also try solve this exclusively for Elixir but then there would be gaps where it wouldn't be supported.

Ecto 2.1 is trying a new approach where the value is configured using a repository callback, that's what we will try to do when Phoenix 1.3 comes out and see where it will lead us to.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

On Sat, Dec 17, 2016 at 1:48 AM, Cory ODaniel <co...@coryodaniel.com> wrote:

I've definitely run into issues where I need to pass an environment variable at run time, but an application that I use doesn't support the {:system, "DATABASE_URL"} or {:system, "PORT"} style environment variables that Ecto and Phoenix support.

I'm curious if it would be beneficial to add support in Application.get_env/2 that when the value that returns matches {:system, var} then System.get_env(var) would be called under the hood.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/9fd6e998-04d2-4d7d-87ee-34abb16ee779%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4%2BOpmZ%3Ds3b2OvAH0chC9wwGL2HrRCB1v2ymbcjOzSc4BA%40mail.gmail.com.

[Allen Madsen]

You can do that for your own stuff. But you can't do it for the libraries you depend on. It requires libraries and frameworks to have the foresight to realize every setting you might want to configure via environment variable. 

Allen Madsen
http://www.allenmadsen.com

On Sun, Dec 18, 2016 at 7:38 AM, Louis Pop <louisp...@gmail.com> wrote:

Why not just use `Application.get_env(app_name, var_name) || System.get_env(var_name)` in your init callback? Seems much clearer to me and requires no additions to get_env.

On Sat, 17 Dec 2016 at 09:17 José Valim <jose....@plataformatec.com.br> wrote:

There has been a couple discussions on the topic either here or on the issues tracker.

The consensus is that this problem needs to be solved but we are not quite sure how. The only way to support {:system, "DATABASE_URL"} in a way that it would also work for Erlang applications is by hijacking the application controller using private APIs. We could also try solve this exclusively for Elixir but then there would be gaps where it wouldn't be supported.

Ecto 2.1 is trying a new approach where the value is configured using a repository callback, that's what we will try to do when Phoenix 1.3 comes out and see where it will lead us to.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

On Sat, Dec 17, 2016 at 1:48 AM, Cory ODaniel <co...@coryodaniel.com> wrote:

I've definitely run into issues where I need to pass an environment variable at run time, but an application that I use doesn't support the {:system, "DATABASE_URL"} or {:system, "PORT"} style environment variables that Ecto and Phoenix support.

I'm curious if it would be beneficial to add support in Application.get_env/2 that when the value that returns matches {:system, var} then System.get_env(var) would be called under the hood.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/9fd6e998-04d2-4d7d-87ee-34abb16ee779%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4%2BOpmZ%3Ds3b2OvAH0chC9wwGL2HrRCB1v2ymbcjOzSc4BA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAM-pwt7n0xP_AQDKX4Qc5DyM2u0B8K5VzEcULD5OCcKsTGaW2Q%40mail.gmail.com.

[Almas Sapargali]

Hello, I'd vote for this change too, actually found that and other discussion, when I was going to make exact same proposal, and just did a quick search first. Then only notable stopper I've seen is that values would be strong, whilist user may expect int/bool etc. I think we could handle this case by passing some typing info in first element, like {:system_int, "POOL_SIZE"} etc, which would evaluate to nil if env isn't set or not an integer.

On Saturday, December 17, 2016 at 3:17:35 PM UTC+6, José Valim wrote:
> There has been a couple discussions on the topic either here or on the issues tracker.
>
>
> The consensus is that this problem needs to be solved but we are not quite sure how. The only way to support {:system, "DATABASE_URL"} in a way that it would also work for Erlang applications is by hijacking the application controller using private APIs. We could also try solve this exclusively for Elixir but then there would be gaps where it wouldn't be supported.
>
>
> Ecto 2.1 is trying a new approach where the value is configured using a repository callback, that's what we will try to do when Phoenix 1.3 comes out and see where it will lead us to.
>
>
>
>
>
>
>
>
>
>
>
> José Valim
>
> www.plataformatec.com.br
> Skype: jv.ptec
> Founder and Director of R&D
>
>
> On Sat, Dec 17, 2016 at 1:48 AM, Cory ODaniel <co...@coryodaniel.com> wrote:
>
> I've definitely run into issues where I need to pass an environment variable at run time, but an application that I use doesn't support the {:system, "DATABASE_URL"} or {:system, "PORT"} style environment variables that Ecto and Phoenix support.
>
>
> I'm curious if it would be beneficial to add support in Application.get_env/2 that when the value that returns matches {:system, var} then System.get_env(var) would be called under the hood.
>
>
>
>
>
>
> --
>
> You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
>

> To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

[Christian Nelson]

The version of this issue I ran into today is with a 3rd party plug, https://github.com/remiprev/plug_canonical_host. I wanted to do something like this:

defmodule HalfDome.Endpoint do
  use Phoenix.Endpoint, otp_app: :half_dome

  plug PlugCanonicalHost, canonical_host: System.get_env("CANONICAL_HOST")

  #...
end

But, between the compilation of the file and the "plug" macro expansion, System.get_env("CANONICAL_HOST") is resolved at compile time. I'm pretty new to Elixir and I tried 4-5 things to have the lookup be dynamic without any luck.

So, I submitted a pull request to that library so that it supports the {:system, "CANONICAL_HOST"} syntax. It was easy to do and works. Unfortunately we have other situations where we want to do the same thing and it's not easy to patch every library that's out there.

Also, I noticed that in the phoenix code, many instances of {:system, env_var} are flagged for deprecation in 1.4... which makes me think trying to perpetuate this convention could be a bad idea.

Is there some elixir-fu that would make it easy(ish) to fetch the CANONICAL_HOST at runtime in the example above?

Thanks!

Christian

[Almas Sapargali]

You can take a look at https://github.com/bitwalker/conform, it’s like config.exs, but captures env variables during application start, unlike config.exs which captures during compilation.

You received this message because you are subscribed to a topic in the Google Groups "elixir-lang-core" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elixir-lang-core/GwR7b9OVRcc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/964e79d1-2b4f-4bc4-ae75-55c18ed42538%40googlegroups.com.

[Eric Meadows-Jönsson]

In this case it wouldn't help to change Application.get_env because the plug function you are using it doesn't actually call that function. I am afraid you would still have to patch every plug library using this kind of configuration.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/964e79d1-2b4f-4bc4-ae75-55c18ed42538%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Eric Meadows-Jönsson

[José Valim]

In your case, the simplest thing to do is to move the plug initialization and execution to runtime:

defmodule HalfDome.Endpoint do

  use Phoenix.Endpoint, otp_app: :half_dome

  plug :canonical_host

  defp canonical_host(conn, _opts) do

    opts = Plug.CanonicalHost.init(canonical_host: System.get_env("CANONICAL_HOST"))

    PlugCanonicalHost.call(conn, opts)

  end

end

Now you will properly read the system environment every time and there is no need for {:system, "something"}.

The {:system, _} tuple environment has been a workaround and right now we have better solutions to the problem. I have written about how Ecto and Phoenix, which provide stateful features, are tackling this problem here: https://elixirforum.com/t/what-is-the-difference-between-using-system-port-and-system-get-env-port-in-deployment/1975/12

However, if the other application always reads from Application.get_env, then you can simply set the application environment directly with the desired above when your app starts:

Application.put_env(:plug_canonical_host, :canonical_host, System.get_env("CANONICAL_HOST"))

To sum the new approach up: runtime/dynamic configuration needs to be done at runtime, and not in the config files.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

On Thu, Mar 2, 2017 at 2:26 AM, Christian Nelson <chri...@carbonfive.com> wrote:

[Christian Nelson]

José,

Thank you... your solution worked perfectly for the plug.

It does seem that the combination of being compiled + macros makes for some tricky work arounds. I have another instance in our router.ex with a forward which I haven't looked into how to work around. If something is just compiled, the lookup can happen in a function. But when it's compiled and a macro, it gets tricky with some APIs. We lucked out that plug :atom exists to invoke a function, as you recommended. I don't think that would work for forward (honestly haven't dug into it yet, so maybe?).

I just read the thread "What is the difference between using..." Something like on_init: {Demo.Web.Endpoint, :load_from_system_env, []} is NOT an alternative solution to my PlugCanonicalHost question above, right?

I appreciate your help and admit I'm still getting my head around how best to configure our Phoenix app, especially since we are most comfortable with the 12-factor app style of configuration, which recommends loading from the env. Additionally, we're deploying to Heroku and using pipelines to promote our stage slug (compiled app) to production. Same code, different configuration.

Cheers,

Christian

[José Valim]

> I just read the thread "What is the difference between using..." Something like on_init: {Demo.Web.Endpoint, :load_from_system_env, []} is NOT an alternative solution to my PlugCanonicalHost question above, right?

The reason why Phoenix and Ecto needs the on_init callback is because they are stateful. If you change the HTTP port, Phoenix cannot magically start listening on a new port. You would need to bring up a new set of processes and shut the old ones down. That's why you need an explicit hook in its lifecycle.

For cases such as PlugCanonicalHost, that can always read from Application.get_env or accept the values as options, so no need for this "indirection".

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/dd63b674-35ed-4afe-833b-4080f197160b%40googlegroups.com.