Mix commands inside iex

[Joe Armstrong]

Is it possible to do mix commands from inside the iex shell? 

It seems that mix commands can only be issued from an external

shell (bash, or whatever) - but not from inside an iex session.

Doing this inside the iex shell would have two advantages:

   - mix would only be loaded once, which would make it respond

     quicker

   - Tab completion would work

     Mix c<tab> could be auto expanded into a mix compile command

Cheers

/Joe

[Paulo Almeida]

It's not as straightforward as calling from the external shell, but it's possible to call Mix.Task.run from iex. Some tasks assume there is a project loaded, so they will only work if iex is started from the root of an existing project.

Inside the root dir of an existing project, run iex with "-S mix" to start the mix application

$ iex -S mix

iex(1)> Mix.Task.run "help", []

iex(2)> Mix.Task.reenable "compile.elixir"

iex(3)> Mix.Task.run "compile.elixir", ["--force", "--ignore-module-conflict", "."]

The implementation of mix tasks is here: https://github.com/elixir-lang/elixir/tree/master/lib/mix/lib/mix/tasks

Regarding the auto-completion of mix tasks, this could also be added to the external shell. At least bash and zsh provide extension mechanisms for auto-completion of command arguments.

[José Valim]

I would like to invite someone to play with some mix tasks in IEx. For example, we should be able to provide something like mix :reload that would stop the currently running app, recompile the code, and start it again. It should be a good workflow for development.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Lead Developer

--
You received this message because you are subscribed to the Google Groups "elixir-lang-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-ta...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Levi Aul]

I was doing just that yesterday, and ended up with https://github.com/tsutsu/mixmux, a toy library for live-session dependency management (probably not the best solution for that problem, but it does what it says.) There were quite a few wrinkles I encountered:

- Re-enabling compilation recursively for an umbrella project that might include tasks other than "compile.elixir" proved far more complex than it should be. I ended up going with Mix.TasksServer.clear rather than dealing with recursing into deps and trying to figure out what backends they used so I could reenable them. It'd be great if there was a Mix utility function you could pass a closure to, that would turn off task-recording within the closure. (If there is one, I never found it.) Or, better yet, if task recording was off by default, and there was a utility function to turn it on, which the Mix CLI would then use.

- There's no API for mutating the state of a loaded Mix project, nor is there a way to trigger Mix to dispose of its internal state and call the methods on the loaded Mix.Project module over again to regenerate its internal state. The first thing I tried in my experimentation was to just define a deps function in my base Mix project that varied with some app-env variable, mutate that variable, and then ask Mix about the project. Mix wouldn't see the change, and there was nothing I could do to make it, beyond popping the project off the stack and then doing a Code.load_file("mix.exs") again to get it pushed back on. A reload method would be very helpful for this. (Though connecting it to triggering compilation might not be the best idea; presuming multiple active IEx sessions, you don't want multiple overlapping compile tasks running accidentally. Better that compilation is explicit, and mix :reload just do something like reloading modules that may or may not have been recompiled, like a more unilateral version of Reprise.)

- I notice that a lot of Mix tasks have "business-logic" code that exists only within the task (deps.compile is annoying for this, but it's not that bad compared to some libraries like Exrm, where the entire release process lives inside the release task module), and where every function other than run/1 is private. This seems like badly-designed MVC code, where code that should be in modules is all in the "controllers" and only accepts webbish JSON-serialized arguments, rather than native structures. Mix tasks should just deal with Mix problems (command-line parsing, etc.), and be coupled with plain library modules that do real work, providing an interface I can call myself to avoid the Mix task entirely.

- Hex breaks terribly when you do certain things that are required in more complex Mix operations (like popping a project so you end up with an empty Mix.ProjectStack, and then adding a new project and calling more tasks.) I had to dig into Hex's state to remove a reference it keeps to some Mix-internal state (that gets destroyed, presumably) so it would continue to operate.

A bit more on the Mixmux experiment:

I was trying to work toward allowing for zero-install applications. With a bit more work, you'd be able to call Application.start with a Mix dep-spec as an argument, and if the application wasn't already available, it'd be retrieved first. If it was available, but had a newer version available matching your dep-spec, it'd replace the app with the new version before starting it.

Built on top of such a zero-install capability, and given OTP, I could expose something like Application.upgrade(new_depspec), which could download the new version of an application, and then do a hot-upgrade from the running version right into it. The only additional work this would require would be getting "upgrade path" metadata exposed somehow, so you wouldn't have to retrieve and hot-upgrade through every version in between v1 and v220 of a package to allow a jump from one to the other.

Assuming a live-hot-upgrade call, I could then build something like an OS package-manager's automatic security update mechanism. Give every Mix.Project an update_url/0 callback, which is supposed to return e.g. an HTTPS RSS %URI{}. (Presumably, for Hex packages, this would point back to Hex.) Have an optional Mix server that watches those URIs, and when it sees new versions announced, Application.upgrade's to them.

And then, since my running node's release would now be changing over time, I would probably want to persist the current application dep-spec list, and possibly the current app-env settings, to disk when they're changed, and allow them to override whatever's in mix.exs and config/config.exs on node startup. I could throw this data in together with mix.lock, and call the result "mix.state". In effect, it'd act sort of like a .boot file for resuming the VM when you do a "mix run" or an "iex -S mix". It'd also end up as the canonical state of any release generated from the project. This would, in effect, mean that you could start an empty IEx session, "compose" a new mix project by installing and configuring deps, and then generate a release from it, all without opening an editor. Sort of Smalltalk-y.

[Eric Meadows-Jönsson]

• Re-enabling compilation recursively for an umbrella project that might include tasks other than “compile.elixir” proved far more complex than it should be.

Re-enabling compiler tasks is tricky because they are dynamic in nature and are different for each project. What we really need is a way to run a task without checking if it was previously ran.

There’s no API for mutating the state of a loaded Mix project, nor is there a way to trigger Mix to dispose of its internal state and call the methods on the loaded Mix.Project module over again to regenerate its internal state.

I’m not sure allowing this would be a good idea, we can’t expect to be able to reload everything in Mix. If you change your mix.exs file it would be better to just restart your iex session.

I notice that a lot of Mix tasks have “business-logic” code that exists only within the task (deps.compile is annoying for this, but it’s not that bad compared to some libraries like Exrm, where the entire release process lives inside the release task module), and where every function other than run/1 is private.

Good criticism, we should be better in this regard. Keep in mind though that we may not want to expose all of the task internals. Updating tasks without breaking backwards compatibility would become a lot harder.

Hex breaks terribly when you do certain things that are required in more complex Mix operations (like popping a project so you end up with an empty Mix.ProjectStack, and then adding a new project and calling more tasks.) I had to dig into Hex’s state to remove a reference it keeps to some Mix-internal state (that gets destroyed, presumably) so it would continue to operate.

Can you explain a bit more what causes Hex to break?

Built on top of such a zero-install capability, and given OTP, I could expose something like Application.upgrade(new_depspec), which could download the new version of an application, and then do a hot-upgrade from the running version right into it.

Except for the simplest OTP applications, doing an hot code upgrade which just reloads modules would certainly cause the application to crash. To properly upgrade an application you need an appup file with instructions on how to do it and any processes that changes it state would need code to upgrade the state. Very few libraries provide the necessary things to upgrade themselves. Upgrades are very complicated and need extensive testing, you should only do them if you really have to. Automating them without writing application specific code is impossible.

​

--
Eric Meadows-Jönsson

[José Valim]

• Re-enabling compilation recursively for an umbrella project that might include tasks other than “compile.elixir” proved far more complex than it should be.

Re-enabling compiler tasks is tricky because they are dynamic in nature and are different for each project. What we really need is a way to run a task without checking if it was previously ran.

This is actually tricky because what if a task that you want to run without checking calls a task that was previously executed too? Should this other task run again too? What if this other task is called multiple times?

I think just cleaning all tasks executed so far is the best way to go.

[Levi Aul]

> I’m not sure allowing this would be a good idea, we can’t expect to be able to reload everything in Mix. If you change your mix.exs file it would be better to just restart your iex session.

You said "restart your iex session", but I think you mean "reboot your node." Just killing the IEx job and starting another one won't reload the Mix.Project. But you could just restart the Mix application, no? Another iex helper, for "restart the :mix application and then reload into it the same Mixfile that was loaded on node startup" would be useful.

> Can you explain a bit more what causes Hex to break?

If you remove the call to Application.delete_env(:hex, :registry_tid) from my toy project, Hex crashes when Mix tries to use it in Mix.Dep.Fetcher after a sequence of uninstalling a loaded dep, and then installing it again. If I recall, it's crashing on the Hex.Registry.stop/0 that happens at the end of the fetch. For some reason :registry_tid is still set, but the ETS table to which it refers has already been released.

> Very few libraries provide the necessary things to upgrade themselves. Upgrades are very complicated and need extensive testing, you should only do them if you really have to. Automating them without writing application specific code is impossible.

Not all upgrades require the complex OTP "freeze the process and send code_change down the supervisor tree" facility that people associate with Erlang's hot code-reloading. There are other ways to automatically upgrade an app: you can make the new version active at next node boot; you can fully stop the old version before starting the new version; or you can just load the new modules online on top of the running app, like you'd do with iex's r() helper. The package author could add Mix or Hex metadata to each release of their package, saying which of those methods (or the code_change one) would work to upgrade to that release, if any.

Then, for any given deployed Erlang system, one or more of the upgrade methods above might be workable. (There's no disadvantages to the online method, for app releases that only need that much. On-next-node-boot upgrades always make sense unless you're running an ephemeral node—they're effectively what Mix does already. The stop-and-start upgrades work fine if you're either a non-realtime system or a Highly Available one. etc.) The person deploying the Erlang release would enable some subset of the upgrade methods based on the operational requirements of the node.

If left with no viable auto-upgrade path, the auto-upgrade process could still make itself useful by logging the fact that some library has now become out-of-date. For something marked as a critical security update, it could even set a SASL alarm until someone comes and does the upgrade manually. (Or removes the offending package.)

(And, if all this seems a bit scary because "releases succeed or fail as a whole", you could always put a CI system in between, attempting the appups and then converting the non-crashy ones into relups, which your nodes could then subscribe to instead of the authors' appups. A bit like Microsoft's WSUS.)

--
You received this message because you are subscribed to a topic in the Google Groups "elixir-lang-talk" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elixir-lang-talk/T2RUVsI5v4k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elixir-lang-ta...@googlegroups.com.

[Eric Meadows-Jönsson]

If you remove the call to Application.delete_env(:hex, :registry_tid) from my toy project, Hex crashes when Mix tries to use it in Mix.Dep.Fetcher after a sequence of uninstalling a loaded dep, and then installing it again. If I recall, it’s crashing on the Hex.Registry.stop/0 that happens at the end of the fetch. For some reason :registry_tid is still set, but the ETS table to which it refers has already been released.

The registry is never stopped after it’s started unless we are trying to update it. There should be no need to delete env vars. Can you open an issue on Hex with the stacktrace and explanation what you are doing to cause the crash.

​