Mix.env and path dependencies

[pragdave]

tldr; The current way that environments are handled encourages the development of monolithic apps. I’d like to start a discussion about this.

Background

The dependencies of an “application/project” are compiled in the :prod environment, regardless of the environment in which the host code is being compiled. This applies to mix compile, mix hex.publish and so on—anything that looks at the project as a whole.

If the developer is working in just that one mix project, then this is fine: the dependencies are basically 3rd party libraries, and are used as-is.

But say the developer (eg me) wants to decouple a monolithic application into a number of components, one mix project per component. When I do this, I have configuration information that is environment specific: paths, debug levels, and even dependencies. But now I have a problem: a component tests OK when run in isolation, but when used as a dependency of another component via a path:, it fails, because it is no longer running in a development config.

For example, we might have the following dependencies in two components' mix.exs files.

component 1:

def deps(:prod), do: [ component2: ">= 0.0.0" ]

def deps(_), do: [ { :component2, path: "../component2" }]

component2:

def deps(:prod), do: [ component3: ">= 0.0.0" ]

def deps(_), do: [ { :component3, path: "../component3" }]

If we compile component 1, then component2 will be compiled in :prod mode, and it will look for component3 in hex, and not locally. This will either fail, or it will use an out-of-date version of component3 (assuming we're currently working on all three components). 

You might say the answer is umbrellas, but I would disagree. I want to champion looser coupling than that (but that's a diversion away from the topic of this post)

Proposal

I suspect the only common use of path dependencies is the one I describe: someone developing two or more components in parallel.

So, could mix be changed so that path dependencies are compiled in the same environment as the project that uses them? I can't see this causing any problems, as they are already treated very differently than the other dependency types. This would automatically fix any use of environment-sensitive values during development and testing.

I'd be happy to create a PR if this sounds like a good idea? I'd be sure to update the docs.

Cheers

Dave

​

[Connor Rigby]

For what it's worth I develop my large Elixir projects this way, as do many Nerves Project developers. You can use the option env: Mix.env() I'm your mix dependency tupple to partially solve this. You don't get iex recompiles either which would nice.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/b1d5c3a6-f3c5-4443-b5c6-a6f10aea6d55%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[José Valim]

> You can use the option env: Mix.env() I'm your mix dependency tupple to partially solve this. You don't get iex recompiles either which would nice.

Yes, passing env: Mix.env() is exactly what umbrellas do too. I think iex recompiles could be made to work for paths and umbrellas (since umbrellas are path deps) by calling "deps.compile" in IEx.Helpers.recompile().

My concern with making env: Mix.env() the default is that I think it would only get halfway of where you want Dave, since the configuration files in the path dependency won't be imported*, and most projects likely need their configuration to run in development. What is your approach here Dave?

*Not sharing configuration is by design since configuration is a global storage. If we simply imported configuration from dependencies, it means any dependency could do "config :kernel, :some_vm_setting, :value", and that could drastically affect your application behaviour - and finding the root cause would be rather tricky.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

[pragdave]

On Monday, July 16, 2018 at 2:04:29 AM UTC-5, José Valim wrote:

> You can use the option env: Mix.env() I'm your mix dependency tupple to partially solve this. You don't get iex recompiles either which would nice.

Yes, passing env: Mix.env() is exactly what umbrellas do too. I think iex recompiles could be made to work for paths and umbrellas (since umbrellas are path deps) by calling "deps.compile" in IEx.Helpers.recompile().

If we could make this the default, I feel it would be a big help, at least for the stuff I'm going to be advocating. (Which may be a good reason not to do it... :)

 

\*Not sharing configuration is by design since configuration is a global storage. If we simply imported configuration from dependencies, it means any dependency could do "config :kernel, :some_vm_setting, :value", and that could drastically affect your application behaviour - and finding the root cause would be rather tricky.

Agreed, but doesn't that paragraph ring alarm bells? There's way too much global stuff already in OTP. And every time you have something global, you introduce yet more forces that drive code towards coupled monoliths. 

I think the main problem with OTP configuration is that it is static. This means that we have an all or nothing situation, where the "application" manages _alll_ the configuration for its dependencies.

In my current stuff, I make configuration active. Each component has a default configuration.It also has, as part of its lifecycle, a place where it gets kicked off by components that use it. During that kickoff, it gets passed overrides to its configuration, which it can then use to update its internal state.

More specifically: each component has a main process, which is not started when that component is used as a dependency. Let's say component One depends on component Two. Both OTP apps get started, and Two establishes its local configuration. It then starts One, passing in any configuration overrides. One then uses these to configure itself, and starts running.s

Currently, I do this by convention: every component defines a struct that represents its configuration, along with default values. Then its main process is started, it gets passed the config overrides. It applies these to the state, and then carries the result forward to every subsequent call. The nice thing about this is that it means you also get runtime reconfiguration for (almost) free.

In an ideal world, the defaults for each components configuration  would come from its local config.exs, but I felt that doing this right now would be really confusing, given that folks currently expect those config values to be ignored. 

I would really like us to move away from using the OTP config mechanism for anything apart from legacy stuff, and to move towards a proper, hierarchical, dynamic way of handling config that encourages a component-based approach to development. In this world, we'd probably delegate config to a separate config server.

But that's a ways away. Right now I'm just thinking about dependencies.

How about it? Can we make Mix.rnv propagate to path dependencies???

Dave

[José Valim]

Dave, I completely agree with your configuration assessment.

The global config rings alarm bells and folks tend to over rely on it. This has been a hot topic during the last year and unfortunately we are still ways away of a solution.

> How about it? Can we make Mix.rnv propagate to path dependencies???

To answer an earlier question, I think there is another case where someone would use a path dependency.

Imagine that you are using dependency X and that dependency has a bug. You want to fix that bug, that usually means:

1. Closing the repository you want to fix the bug

2. Fixing the bug

3. Verifying that the bug is fixed by depending on the local dependency using :path

If we default to Mix.env, it means you would be testing that dependency in development, which is not the environment that you were using or where you would use it. In other words, there are scenarios where you would use third-party dependencies as path dependencies. The configuration mismatch means hard to debug errors may also arise from this situation (which is why I asked about configs).

So my answer would be a no: I wouldn't make Mix.env() propagate by default because of third party dependencies. I would rather require it to propagate explicitly via env: Mix.env().

However, I do have a counter proposal: the "in_umbrella: true" has NOTHING about umbrellas in its behaviour. Here is the source: https://github.com/elixir-lang/elixir/blob/894ce4e305855c600a528f91872c66e4958ca788/lib/mix/lib/mix/scm/path.ex#L22-L28

Maybe we could find a better name for it that would work both inside and outside of umbrellas? We could call it "sibling: true" but I find too vague. Thoughts?

> If we could make this [mix deps.compile in recompile] the default, I feel it would be a big help, at least for the stuff I'm going to be advocating

We can totally call "mix deps.compile" in IEx.Helpers.recompile. Can somebody submit a pull request? When testing it manually, please make sure that you can recompile the same path dependency multiple times. Basically this:

1. start parent app

2. call recompile()

3. change path dep

4. call recompile() and check it works

5. change path dep

6. call recompile() and check it worked again

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/3142851f-07d8-48bc-8a00-304feae470f1%40googlegroups.com.

[Connor Rigby]

I just tried adding a quick Mix.Task.run("deps.compile", arguments) in the iex helpers file. It seems to :noop on recompiling path deps with compiler errors. 

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/3142851f-07d8-48bc-8a00-304feae470f1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4Kyt0kq7%3DVC27UV1L5KjaCgaW-kD4%2Bq4qsMt3uAn5Kjzg%40mail.gmail.com.

[José Valim]

Yup. You will have to make sure that deps.compile is reenabled as well as the inner compile task of the local dependency.

--

[pragdave]

To answer an earlier question, I think there is another case where someone would use a path dependency.

Imagine that you are using dependency X and that dependency has a bug. You want to fix that bug, that usually means:

1. Closing the repository you want to fix the bug

2. Fixing the bug

3. Verifying that the bug is fixed by depending on the local dependency using :path

I agree this is a possible case, but I’d argue that (a) it’s rare, and (b) its one of those cases where you’re already changing the dependency, so adding an env: is not a hardship.

But I’m not too fussed about this now that we can alias in_umbrella :)

Maybe we could find a better name for it that would work both inside and outside of umbrellas? We could call it "sibling: true" but I find too vague. Thoughts?

This flag is used to indicate that the dependency is used in the same environment as the dependor. So how about something like:

    same_env:  true
    in_my_env: true

​

[José Valim]

Dave, just to make sure that something did not get lost in the way.

:in_umbrella today means more than "same_env". The following:

{:foo, in_umbrella: true}

Means:

{:foo, path: "../foo", env: Mix.env()}

In other words, the feature that you are asking for is already available today. You need to pass "env: Mix.env()" for your dependencies. I personally find "env: Mix.env()" clearer than "same_env: true". That's because "same_env:" does not tell me which env is it going to be the same to, while "env: Mix.env()" tells me I want the dependency in the same environment as the current one. What are your thoughts?

In the previous e-mail I have suggested a replacement for :in_umbrella, which means :same_env is not enough since :in_umbrella does a bit more than that.

Thoughts?

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

--

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/13efe6fc-3a77-4ea4-8a48-4a91680802b3%40googlegroups.com.

[pragdave]

On Monday, July 16, 2018 at 3:11:25 PM UTC-5, José Valim wrote:

Dave, just to make sure that something did not get lost in the way.

:in_umbrella today means more than "same_env". The following:

{:foo, in_umbrella: true}

Means:

{:foo, path: "../foo", env: Mix.env()}

OK, I didn’t think about that aspect.

How about is_peer:, meaning it is treated the same. And make it an alias, so in_umbrella still works.

​

[José Valim]

I am worried that we are introducing a new name without gains. In this case, I would prefer is_sibling, but I find both is_sibling and is_peer vague.

Naming. :(

[Yevhenii Kurtov]

What is the reason to introduce an `is_peers` alias? It doesn't introduce any new functionality nor improves clarity. Isn't it just codebase bloating?

--

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/28457c75-64d3-4f2f-acd4-5fbdba5274ae%40googlegroups.com.