Mix task: patching `mix.exs` file and recompiling

[Zach Daniel]

Hey folks!

In igniter, a core concept we have is `mix igniter.install dep`, which patches the `mix.exs` file, recompiles the app accordingly, and then looks for a mix task called `mix <dep>.install`, and invokes it. There are lots of other features, but this is the crux of it.

In an ideal world, I could patch the `mix.exs` file with a new dependency and run `Mix.Tasks.run("deps.compile")`. 

What we're doing now:

I don't necessarily want to dive too deep into what I'm currently doing, as we do have something working (but far from ideal) but its through a lot of trial and error is a bit of a mess. It also recompiles every dependency every time. I'd much rather revisit this from a fresh perspective.

So with that in mind, what would the recommended way to approach this idea be? 

Specifically, how could I write a mix task that can patch the `mix.exs` file, and then compile the new dependency or dependencies?

[José Valim]

It may be hard to tackle this. For example, adding a dependency may cause other dependencies to be recompiled. That’s ok, but if those dependencies have already been running, then we need to shut down applications, purge modules and state, etc.

Therefore we can make this work for certain cases. It really depends on how much code has been loaded at the time the command is invoked.

Generally speaking, it is easier to make it work if “mix deps.add —install” was part of Elixir, because we could then call it without loading much else. However, I don’t know how hard would be to implement this in practice without porting tools like sourceror. If it is only a small part of it, it would be fine.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/20c0edaf-60ef-464e-80f8-37734c67ce19n%40googlegroups.com.

[Zach Daniel]

So in igniter, we don’t need applications to ever be started, we only need to be able call their mix tasks. We own the entry point (via a mix task), and we can make sure that people are aware that you can’t call these mix tasks while applications are running. So assuming we never start any of the loaded applications, and only need the code to be loaded, does that make things easier? 

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4LFnDMfC7jMrdE5-YgcLt%3DaXbcmEkPQR0O5GZrp2wVF5Q%40mail.gmail.com.

[José Valim]

Partially. I am assuming the goal is to add a new function to Elixir/Mix. But I am not sure if it is worth adding an API that can only be invoked in the exact same scenarios igniter. It definitely feels too narrow.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/4C5CCE37-D2B8-4B9E-8E75-080C2090626B%40gmail.com.

[Zach Daniel]

I see your point, and I’m more than comfortable with it not being added to Elixir or Mix, except for perhaps if we can vet it in the context of `Igniter` and then it proves valuable enough to upstream. I wasn’t thinking to add something in Elixir/Mix that actually mutates the `mix.exs` file, rather something that you could run that would recompile an application fully in light of changes on the file system. That seems far more generally useful to me.

The end goal, however, is for any elixir package to be able to set up igniter installers, it’s not specific to any package or package ecosystem (i.e Ash). Perhaps there is some small adjustment that could be made to `mix deps.compile` that would *allow* for doing this, like a command that tells it to reread the `mix.exs` file, or a command to unload and load a mix project in such away that `mix deps.compile` would “do the right thing”. Maybe the tools I need are already there even, I just haven’t been able to fin them.

Ultimately I’m just looking to figure out the least error-prone way to do this, and have found myself at the edge of my knowledge trying to build this out. With or without changes in Elixir/Mix. We have something that “works", but it required copying and modifying the `mix deps.compile` task implementation, and currently force recompiles every dependency. Not because I think that is the right thing to do, but because it’s the only thing I’ve tried currently that reliably works in `mix igniter.new` and `mix igniter.install`.

For context, `mix igniter.new` is an archive that creates a new project, adds igniter, and then runs `igniter.install`.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4KQ2ZC%2Bu_FmUCeH_SUFz0-YvCA9Z%3DGnUFBhpjK-iEcfZQ%40mail.gmail.com.

[José Valim]

So, just to be clear, my point is that it may be easier to add "mix deps.add --install" to Elixir (i.e. add igniter itself) than adding "a command that tells it to reread the `mix.exs` file, or a command to unload and load a mix project in such away that `mix deps.compile` would “do the right thing”."

Doing the command requires doing a new resolution, figuring out all deps that require add/recompiling/removing, stopping them, unloading them, purging code, and more. And you may still have left-over state somewhere (such as persistent term). The only case doing this is easier is by adding a bunch of exceptions to the rules above, which then only makes it useful for the exact scenarios as igniter (which brings me back to the previous paragraph) or makes it slower (because the VM is optimized for code loading, not code purging).

The counter arguments to the above are:

* We may need to implement the above anyway for Mix.install/2: https://github.com/elixir-lang/elixir/issues/12293

* Language servers need to do something similar to the above and, because Elixir wants to incorporate more language server tools, it may need to implement the above too

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/67039EE1-030A-47A1-BE41-E58A3905EE03%40gmail.com.

[Zach Daniel]

Understood.

There are a lot of interesting complexities in having a `mix deps.add` task, and the process that we’ve built for igniter has quite a few useful components that I think are unlikely to make it into a core `mix deps.add --install`  task. 

We support composing additional required installers, specifying option schemas, composing their arguments, etc. Composiing installers requires multiple phases of user confirmation, i.e “add a dep, run an installer, that installer requires adding another dep”, and synthesizing all of the appropriate changes into one diff at the end (aside from dependency changes). We wouldn’t be able to compose installers if a requirement for making this work is that `mix deps.add —install` is invoked from the CLI. 

If `mix deps.add —install` can be invoked from something like `Mix.Task.run` then I’m pretty sure we’re back in the world where the “reread mix.exs and unload/reload” logic has to exist. So we’d either have to go considerably down in terms of QoL features we’ve built up in igniter.

There are additional things that we are going to tackle soon, specifically if you wanted to support doing something like `mix deps.add package —only test`, the rereading of the mix.exs would not compile that module if your task was run in dev env, and so its installer would not be run. Furthermore, some installers would prefer to express explicitly that they should be added with `runtime: false` and `only: [:test]`. We will handle this in igniter by storing these nested installers and printing them to the user to show them what tasks they have to run, because those requested installations must be run in a different environment, and reinstalling with a —only test flag will patch that into the mix.exs

> run these installers after you’re done

> MIX_ENV=test mix igniter.install test_dep1,test_dep2 —only test

Sorry for the wall of text 😂 Ultimately my vote goes to the command that can be used to reload a project (or whichever strategy is taken). Easy to say, maybe hard to actually make happen. We have something that works “okay” for now, so I’m not pushing for anything in a rush, and I’m more than happy to contribute towards this effort.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4KY%2Bmw01vd0w1vO75sSfePp4fGUTM57kNW3q5%3D-sx2mvg%40mail.gmail.com.