Automatically infer the list of applications

[José Valim]

One of the most confusing topics when it comes to building Mix projects and releases is in figuring out the list of applications. The workflow for adding a dependency usually requires adding the dependency to the list of deps and then adding its application to the list of applications.

Since this was a common source of confusion, we wrote a blog post discussing the common scenarios and rationale: http://blog.plataformatec.com.br/2016/07/understanding-deps-and-applications-in-your-mixfile/

However, I believe we should still attempt to simplify this workflow. This is such a proposal.

The idea is to automatically inflect the list of applications by including all dependencies that are required in production. Overall, we will have:

def application do

  [otp_applications: [:logger],

   skipped_applications: [...],

   included_applications: [...]]

end

   

If the :applications key is is not provided, we will automatically inflect it by calculating:

applications = ((all_apps_for_prod_deps ++ otp_applications) -- included_applications) -- skipped_applications

• otp_applications - lists applications that come from erlang or elixir
• skipped_applications - dependencies that are listed in production but you want to skip during releases (mutually exclusive with :applications)
• included_applications - same meaning as today. applications you want to include in production but not start them

I expect for the majority of cases developers won't need to pass the skipped_applications and included_applications flags, so we can skip them in generated templates. At the end, new apps will have only the following:

def application do
  # List only applications from Elixir and Erlang required in prod

  [otp_applications: [:logger]]

end

Thoughts?

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

[Louis Pilfold]

Hiya

I'm unsure about this proposal. I do think that the name OTP
applications is unclear. There are other Erlang OTP applications that
are started when not in this list, and the skipped and included
applications are OTP applications as well.

How does one determine if a production dependency is an application or not?

Cheers,
Louis

On 20 September 2016 at 10:01, José Valim

> --
> You received this message because you are subscribed to the Google Groups
> "elixir-lang-core" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elixir-lang-co...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4LWOsR5xiND%3D-Ty_NcFmFj3VfPstVs1mxTiTsbrf6g64Q%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

[José Valim]

I do think that the name OTP applications is unclear.

I agree. It has a double meaning: we can call all applications an OTP applications or we can call the ones that ship part of OTP (the standard library) an OTP application. Suggestions for better names? We could call it `kernel_applications` or `stdlib_applications` if that's clearer?

 

How does one determine if a production dependency is an application or not?

Production dependencies must always be applications unless you have a very special case of a production dependency that is compile-time only (and I can't recall of any).

[José Valim]

One other option is to have something called configure_applications (although something shorter would be ideal):

configure_applications: [logger: :start, other: :included, compile_time_only: :skip]

We will automatically include all production deps but you can remove or add something by using configure_applications.

Further thoughts?

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

[José Valim]

Another idea is to focus on the most common case and built on top of that:

  def application do

    # Any application that does not come from a dependency, such

    # as Erlang and Elixir applications must be explicitly listed.

    [start_applications: [:logger]]

  end

But you can also pass ", other: :included, compile_time_only: :skip". The new :start_applications cannot be used alongside the old :applications/:included_applications options.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

[Paul Schoenfelder]

I actually built exactly this feature into Distillery, but then removed it because I didn't have a good solution for omitting build-time only deps. I considered an option on dependencies, i.e. `release: false`, but that didn't feel right, and I didn't want to overload the meaning of things in the applications list.

We can infer the entire list of applications for prod dependencies without having to specify anything in the `:applications` list, but there are two other things we need to be able to specify, application start type (`included_applications` is *not* for this, it is for taking over the application lifecycle of the applications in that list, and only one app in a given release can include an application), and compile-time only deps.

The former is solved by simply setting the start type in the applications list we already have, i.e. `myapp: :load`, this is already supported today - we just need a mechanism for the latter. `skip_applications` seems like a reasonable option to me, flagging deps seems like another, but as I mentioned above, I'm not sure it feels right.

I don't agree that "OTP application" has a double meaning, it has a very clear definition, which is any application that conforms to the OTP structure (i.e. has a supervisor tree). The thing is, regardless of whether an application is an OTP application or simply a library application, it has to be included, so I find the distinction not particularly useful.

Paul

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4Lh%3Dp4bokfCv6KiG5dRGJHqXUkQOyMMogEzGXrfpd2G_w%40mail.gmail.com.

[José Valim]

Thank you Paul for your feedback, it is very valuable for this discussion.

Once we consider releases, there are at least three level of configurations happening here:

1. dependencies (which are mostly about getting the dependencies sources to your system

2. applications

3. releases

James also proposed something related to your dependencies list but we discarded the idea because we also didn't feel it was correct. So far dependencies are about getting the sources into your system and compiling them to applications.

James also proposed some features that would require us to change how Mix starts applications and while at first we decided to not do that, I now realize it may be worth pursuing since it will help bring Mix and releases together.

In other words, it may be worth looking for a mechanism that allows us to specify the application type directly in def application and make sure both releases and Mix respect those types. 

My only concern is that, although it may be compelling to bring applications and releases metadata together, releases may have multiple targets and therefore may still need to add their own configuration. So it is important whatever format we choose can be used for both applications and releases.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAK%3D%2B-Tt7jDioik%2B9AwhHca8grUkuKdiHi9%2BgNSzE0%2BofKyDvjw%40mail.gmail.com.

[Paul Schoenfelder]

James also proposed something related to your dependencies list but we discarded the idea because we also didn't feel it was correct. So far dependencies are about getting the sources into your system and compiling them to applications.

You mean the implementation from Distillery which automatically constructed the applications list? Yeah it was flawed because it always included dependencies, even if they were compile-time only. I do think there is generally a 1:1 correlation with prod dependencies and applications which should be part of the release, but there are obviously exceptions, Distillery itself being one - this was ultimately why I removed the automatic inference. Sadly there doesn't seem to be a clear way to automatically determine what applications are only used at compile-time vs runtime.

James also proposed some features that would require us to change how Mix starts applications and while at first we decided to not do that, I now realize it may be worth pursuing since it will help bring Mix and releases together.

Could you provide some details on that, or link to the previous discussion? I'd be interested in reading about the proposed change just for the context.

In other words, it may be worth looking for a mechanism that allows us to specify the application type directly in def application and make sure both releases and Mix respect those types. 

I like this idea, though without more information I'm not precisely sure what is meant by "application type" here. Do you mean that in `def application`, one would flag whether their app is a compile-time vs. runtime application?

My only concern is that, although it may be compelling to bring applications and releases metadata together, releases may have multiple targets and therefore may still need to add their own configuration. So it is important whatever format we choose can be used for both applications and releases.

I think release configuration can (or should) be viewed as a superset of application configuration - in other words, releases build on the application configuration, they don't differ from it. The configuration for a specific release target shouldn't countermand the configuration for the application(s) it's based on. To be clear for everyone reading, by "application configuration", I don't mean runtime config stuff that goes in `config.exs`, but rather the application definition from `mix.exs`. But perhaps some specific real-world examples may show where this falls down - so far in my experience, the above holds true.

Paul

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4%2BAd5V5ga6sF1MkyC8zCo7WDRr4NFNMFNvmfKNKG6W1jg%40mail.gmail.com.

[José Valim]

You mean the implementation from Distillery which automatically constructed the applications list?

Sorry, I meant your idea of adding release information to the list of dependencies.

 

Could you provide some details on that, or link to the previous discussion? I'd be interested in reading about the proposed change just for the context.

It was a very quick exchange in private on IRC. Basically we want to support something like start_applications: [foo: :load], as you proposed later on, but we decided to not do that because we would need to change how Mix starts applications (today it starts applications by mostly calling Application.ensure_all_started/2). However, if the goal is to make it closer to releases, then the work may be worth it.

 

I like this idea, though without more information I'm not precisely sure what is meant by "application type" here. Do you mean that in `def application`, one would flag whether their app is a compile-time vs. runtime application?

We are discussing two types of application configuration (hence the confusion)

1. If it is temporary, transient or permanent

2. If it should be started, loaded, included, none or skipped (am I missing any?)

And to agree with your last paragraph: I believe everyone agrees at this point we should infer the applications and that bringing applications and releases together is a good idea. The next question then is: how to make this all possible?

[Paul Schoenfelder]

We are discussing two types of application configuration (hence the confusion)

1. If it is temporary, transient or permanent

2. If it should be started, loaded, included, none or skipped (am I missing any?)

I think it's unnecessarily confusing to split them apart that way, I think a more logical split is like so:

1. {:start, :permanent | :transient | :temporary}, :load, :none - {:start, _} isn't a thing of course, I'm just using it to reflect the grouping, but these are the different choices for loading/starting dependent applications when the top-level application starts

2. :included_applications - this is basically a variant of :load, but worth keeping separate for purposes of this discussion, since it has some caveats vs :load - namely that only *one* application in the entire app tree can "include" a given application. The use of included applications is basically to ensure that the lifecycle for that application is controlled entirely by the including application. It's a horrible, confusing name for what it does, but at the same time I don't have any better suggestions, maybe "owned" instead of "included"? In any case, staying close to the Erlang terminology is probably more valuable here, but it does annoy me.

3. :skip/:skipped_applications/:excluded_applications - Some logical variant of this is basically what we need in order to bring Mix applications and releases together. 

All of the names I just mentioned for #3 however are pretty unintuitive, which I guess is just one hangup here, the other being whether that's really the right way to go about excluding apps. Ultimately, the desire that we have is how to say "this application is not needed at run time for my application, so I don't need it outside of compile-time". Maybe a different start type (:compile) could be used to say "start this at compile-time, but skip it when producing build artifacts (such as releases)". Ideally, we could programmatically determine whether an application is used at runtime, it seems like that's not possible (yet), but it may be a path worth exploring unless there are hard limits on why it doesn't work. One thought I had is that it should be possible to dump imports for a module from the BEAM info and map those back to their owning applications, and using that to determine what apps are truly needed at run-time. Thoughts?

Paul

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4%2B9x%2BVdar9vFcMk_GzjFOnpVe_j8EfBKP2gS9%2BG92g5Uw%40mail.gmail.com.

[José Valim]

After further discussion on IRC, we have raised some concerns about the latest proposal.

The issue with supporting entries such [myapp: :load] or [myapp: :permanent] is what happens when projects A and B specify different properties for :myapp. We would need to solve conflicts and that would introduce complexity which otherwise does not exist on releases because relaeses specify this metadata only once.

Therefore, one last proposal is the following:

def application do
  # Any application that does not come from a dependency, such

       # as Erlang and Elixir applications must be explicitly listed.

  [extra_applications: [:logger]]

end

We will also support the :runtime option (or something more appropriately named) for dependencies that allow us to specify a dependency is not necessary during runtime. For example, distillery itself would be specified as:

{:distillery, ">= 0.0.0", runtime: false}

With the features above, the list of applications, when not specified, will be calculated by:

applications = (all_runtime_apps_from_prod_deps_ ++ extra_applications) -- included_applications

This seems to be the simplest approach conceptually while solving the main problem of requiring users to explicitly specify the applications list.

Final thoughts?

[Paul Schoenfelder]

The issue with supporting entries such [myapp: :load] or [myapp: :permanent] is what happens when projects A and B specify different properties for :myapp. We would need to solve conflicts and that would introduce complexity which otherwise does not exist on releases because relaeses specify this metadata only once.

The conflict resolution for this is actually pretty simple: if A specifies [myapp: :load], and B specifies [myapp: :permanent], B wins, because B cannot start if :myapp is not started, but A can still start if :myapp is both loaded/started. This is how the conflict would have to be resolved for releases, aside from just raising an error and requiring A to be modified to use :permanent as a start type.

Resolving the start type between :permanent, :transient, and :temporary, is simply determining precedence, which is given by that order.

We will also support the :runtime option (or something more appropriately named) for dependencies that allow us to specify a dependency is not necessary during runtime. For example, distillery itself would be specified as:

{:distillery, ">= 0.0.0", runtime: false}

I think this solves the run-time vs compile-time determination, so I'm +1 

 

Therefore, one last proposal is the following:

def application do
  # Any application that does not come from a dependency, such

       # as Erlang and Elixir applications must be explicitly listed.

  [extra_applications: [:logger]]

end

I really feel like adding an extra field here doesn't simplify anything. The algorithm could just as easily be:

applications = applications ++ (all_runtime_apps_from_prod_deps_-- included_applications)

Where any items manually added to the applications list take precedence over those which would be inferred. Is there any reason why adding `extra_applications` is desirable?

Paul

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4JZ%3D%2BUVsx%2BwAUy%3DMaO5%2Bk8oLSb1mbn_7DmS03o9pMeOuw%40mail.gmail.com.

[José Valim]

 

The conflict resolution for this is actually pretty simple: if A specifies [myapp: :load], and B specifies [myapp: :permanent], B wins, because B cannot start if :myapp is not started, but A can still start if :myapp is both loaded/started.

I am afraid it may not be that simple. Because if A requires the application to only be loaded because let's say it needs to set some configuration before the application starts, we would need to make sure myapp is loaded before A and that myapp is started before B. So we need a slightly more complex dependency resolution which gets trickier in cases where B also has a dependency on A.

This is how the conflict would have to be resolved for releases, aside from just raising an error and requiring A to be modified to use :permanent as a start type.

How would this conflict arise inside releases though if the type (permanent, load, etc) is specified in only one place? 

 

I really feel like adding an extra field here doesn't simplify anything. The algorithm could just as easily be:

applications = applications ++ (all_runtime_apps_from_prod_deps_-- included_applications)

I would still like to have a mechanism where we disable the dependency guessing and that mechanism should be backwards compatible hence I think we should keep how :applications work today. Your code above also wouldn't work by default, it would require at least a call to Enum.uniq/2 (which means the application list is even more ambiguous from the user perspective).

[Paul Schoenfelder]

I am afraid it may not be that simple. Because if A requires the application to only be loaded because let's say it needs to set some configuration before the application starts, we would need to make sure myapp is loaded before A and that myapp is started before B. So we need a slightly more complex dependency resolution which gets trickier in cases where B also has a dependency on A.

It's simpler if B has a dependency on A, since it means A has to be started before B can be started, which would likely work out fine. But either way, you can't know when A will start :myapp, or if it ever will, which is why A would need to add :myapp to :included_applications, the .script which would get generated would then load :myapp, load/start A, which would start :myapp in it's top-level supervisor (after configuring it/etc.), then once A (and thus :myapp are started), start B. This would happen with or without the explicit dependency on A by B, since systools resolves the dependency order of applications when it generates the .script for the release.

How would this conflict arise inside releases though if the type (permanent, load, etc) is specified in only one place? 

I mean that is how it would be solved logically if applications could specify their start type requirements for dependent apps - it's effectively what must already be done today, just by hand.

I would still like to have a mechanism where we disable the dependency guessing and that mechanism should be backwards compatible hence I think we should keep how :applications work today. Your code above also wouldn't work by default, it would require at least a call to Enum.uniq/2 (which means the application list is even more ambiguous from the user perspective). 

Well, my code was more pseudo-code, as I mentioned, the manual entries in applications would need to take precedence over anything inferred, which is more complex than a single line of code can express clearly. You do bring up a good point about backwards compatibility though, extending :applications means that it would only work on that version of Mix or greater.

I don't like the idea of "extra_applications" though, the name just doesn't seem to express what the list is really there for - namely to provide metadata about the load/start behaviour of applications. Maybe it's simpler/clearer to break them out, e.g. "loaded_applications", "started_applications", where the latter allows one to specify the start type, defaulting to :permanent. Thoughts?

Paul

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4JoG%2B0pmLcLJt5Xciiw0UmVLAukkT0GLG4u0NOX7cRnPg%40mail.gmail.com.

[OvermindDL1]

This sounds better to me too, `extra_*` has some odd connotations that makes it seem like it does not belong...

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

[José Valim]

It's simpler if B has a dependency on A, since it means A has to be started before B can be started, which would likely work out fine.

Right. :D I meant the opposite.

I will check systools and see how it would handle those cases closely.

 

I don't like the idea of "extra_applications" though, the name just doesn't seem to express what the list is really there for - namely to provide metadata about the load/start behaviour of applications. Maybe it's simpler/clearer to break them out, e.g. "loaded_applications", "started_applications", where the latter allows one to specify the start type, defaulting to :permanent. Thoughts?

The last proposal completely dropped the ability to set an application as permanent, temporary, etc. hence the name :extra_applications. If we are going to have such ability, I would stick with :start_applications previously proposed. 

--

[Paul Schoenfelder]

The last proposal completely dropped the ability to set an application as permanent, temporary, etc. hence the name :extra_applications. If we are going to have such ability, I would stick with :start_applications previously proposed. 

Why drop that capability? It's rare for sure that someone would need a start type other than :permanent, but nevertheless I think it should be supported. Adding :start_applications makes sense (assuming one can specify the start type, otherwise it's not needed and we can just use :applications for this), but what about :load only apps?

Paul

 

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4%2BRcrmMZcfmivW9Hp3sQx_U56%3Dg38OjkonpFQoOv%3DTc5g%40mail.gmail.com.