Proposal: Mix.install

[woj...@wojtekmach.pl]

Today, the only way to use dependencies in Elixir is by first adding them to the current Mix project or installing them as archives. This makes Elixir less than ideal for simple tasks where you'd usually use a scripting language, e.g. download a JSON file, parse it, extract some data. This is also adding pressure for a more "batteries-included" approach in the stdlib to make these simple tasks more convenient to accomplish and require neither creating a Mix project nor installing dependencies.

I'd like to propose adding a `Mix.install` function that will download, compile, and load given dependencies into the VM.

Here's how we could use it from an IEx session:

    $ iex

    iex> Decimal.new(42)

    ** (UndefinedFunctionError) function Decimal.new/1 is undefined (module Decimal is not available)

        Decimal.new(42)

    iex> Mix.install([{:decimal, "~> 2.0"}])

    :ok

    iex> Decimal.new(42)

    #Decimal<42>

Or from a script:

    $ cat run.exs

    Mix.install([

      {:decimal, "~> 2.0"},

      {:jason, "~> 1.0"}

    ])

    IO.puts Jason.encode!(Decimal.new(42))

    $ elixir run.exs

    "42"

You need to call `Mix.install` in your VM and explicitly list all your dependencies. And if you start a new VM, you'd have to call `Mix.install` again. 

Under the hood, the `Mix.install` starts Mix and creates a temporary Mix project to install and load the dependencies and unloads the temporary project when it's done. For simplicity, the initial implementation allows you to only call this function outside of a Mix project and you can only call it once in a given VM.

The temporary project is preserved across runs so if you start another VM and call `Mix.install` with the same set of dependencies, they'll be loaded right away without needing to download or compile them. Each set of dependencies will have it's own temporary directory, roughly: `/tmp/mix_installs/<md5_of_deps>`.

The location and the layout of the temporary project is considered an implementation detail that users shouldn't depend on.

The fact that the resulting `mix.lock` file is hidden from the users presents challenges around reproducibility and updating deps. If you pin your deps to specific versions, a call to `Mix.install` will give consistent results, however if you use version requirements like `~> 1.0`, it may give different results depending on what the deps resolve to at at given time. Similarly, if there's a new version of a package that satisfies a given version requirement, `Mix.install` won't automatically install it as it already cached the dependency. For this, you'll be able to pass a `force: true` flag to bypass the cache and install & compile from scratch for a given set of dependencies. This will also help if your install somehow got into an inconsistent state.

### Reproducibility

Another way of quickly installing and using dependencies is via workspaces, you'd install and automatically load dependencies within a given workspace. That brings questions about reproducibility.

Imagine you write a piece of Elixir code that uses dependencies and share it with people, how reproducible is it?

- using Mix project, the dependencies are fully reproducible given the Mix project maintains a `mix.lock` file

- using `Mix.install`, the dependencies are reproducible within the specified version requirements

- using global state like workspaces, the dependencies are not reproducible. If you ask a user to run your script, you need to tell them what dependencies they need to install

Between global dependencies having very weak reproducibility and Mix projects being fully reproducible, `Mix.install` is a compromise where you can reasonably reproduce things within a list of requirements. 

### Further reading

Thanks to https://hex.pm/packages/teex authors for exploring some of these ideas!

Also see https://github.com/wojtekmach/playground for an earlier proof-of-concept and https://github.com/elixir-lang/elixir/compare/master...wojtekmach:wm-mix-install for a work-in-progress implementation.

[José Valim]

I have discussed this with Wojtek and I am in favor of this proposal.

Elixir was designed to support scripting but I constantly hear that the lack of some quality of life features, such as trapping exits, limited IO handling, lack of support for dependencies, etc make it so even advanced Elixir developers don't end-up using Elixir for regular single-file scripts.

I have never been a fan of global dependencies. Between global dependencies (not easily reproducible and with implicit dependencies) and Mix projects (fully reproducible with explicit dependencies), this sounds like a reasonable compromise (explicit dependencies reproducible within requirements). The only downside of this approach compared to global dependencies is that each IEx session is a blank session - but that can be streamlined in the future.

Note the goal here is not to replace escripts either. Escripts have the full-backing of a project, with dependencies, multiple source files and tests. If it is anything beyond experimentation or more than a couple dozen lines, then you do want a complete project.

TL;DR: +1

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/ed320491-005c-4414-b1a9-5ca0e540bd3dn%40googlegroups.com.

[woj...@wojtekmach.pl]

A minor addition: I'd like to add a shortcut so you can write `Mix.install([:decimal])` instead of  `Mix.install([{:decimal, ">= 0.0.0"}])`.

> The only downside of this approach compared to global dependencies is that each IEx session is a blank session - but that can be streamlined in the future.

Agreed. I didn't mention this in the proposal but here's a trick I've been using:

# ~/.iex.exs

unless Code.ensure_loaded?(Mix) do

  Mix.install([

    :decimal,

    :jason,

    :nimble_csv,

    {:requests, github: "wojtekmach/requests"}

  ])

end

This means however that when you start a session, you won't be able to call `Mix.install` again. In those cases I'd `iex --dot-iex ""` to ignore my global config.

[woj...@wojtekmach.pl]

Oops, this is the correct `~/.iex.exs`. The idea is I want to use Mix.install when using `iex` but _not_ (as it will fail) when using `iex -S mix`.

if :mix not in Enum.map(Application.started_applications(), &elem(&1, 0)) do

  Mix.install([

    :decimal,

    :jason,

    :nimble_csv,

    {:requests, github: "wojtekmach/requests"}

  ])

end

[eksperimental]

`Mix.install(:decimal)` should also work out of the box IMO.

> >> <https://groups.google.com/d/msgid/elixir-lang-core/ed320491-005c-4414-b1a9-5ca0e540bd3dn%40googlegroups.com?utm_medium=email&utm_source=footer>
> >> .
> >>
> >
>

[Jon Rowe]

I plus one this proposal, it’s similar (for scripts at least) to what Ruby has with bundler inline mode, I like this because such techniques allow you to create single file reproductions of issues, which are great for bug reports in PRs.

Cheers

Jon

----------------

ma...@jonrowe.co.uk

https://jonrowe.co.uk

[Felipe Stival]

+1 for this proposal. 

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/dejalu-217-a7302266-156c-46d8-86f9-089e2065af2b%40jonrowe.co.uk.

[Anil Kulkarni]

I love the general idea. I think it's a great solution for copy/paste/run needs that a scripting language should have. 

I do have a question about the caching packages across runs. What is the motivation? Is it to increase performance, or are there other reasons? 

My concern is that inevitably there will be issues with the cache. It reminds me of global pip requirements, global npm installs etc. 

Some issues I have personally run into with these solutions ( python and js) include: 

1) the cache size increasing and needing to manually trim it. Consider popping a shell into a production environment to fix an issue. You wouldn'twant to leave stuff behind. 

2) Managing dependencies across language upgrades (e.g. updating node to 15.5, what happens to the dependencies? What happens for sxs (side by side)

3. Inconsistencies due to caching, especially with node. E.g. Needing to blow away the cache and download fresh to fix an issue 

4. Fire corruption due to other programs needing to nuke the cache 

5. Issues that arise when backing up and then trying to restore to a different home folder (I had a python dependency hard code the path an install)

6. File permission changes, especially when switching between different users on chroot'd terminals 

Some of these issues are going to be a problem anyways, but I think that caching provides another avenue  for things go wrong.

My suggestion would be to either add explicit cache management functionality to elixir - including which path it lives, and blowing away the cache or to not use one for now.

The former could look like this: 

ELIXIR_CACHE_DIR=/foo iex 

(eg set an environment var then call iex) or with parameters passed in iex --cache-dir 

iex --clear-cache 

The latter solution could most easily be solved by using /tmp and making a new folder each iex instance that calls mix install (possibly also respecting the cache dir flag options)

-Anil

On Jan 21, 2021, at 05:02, Felipe Stival <v0i...@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAKC64%2ByRtG-XGWkSADPXHzOLZ8AJ4g73DxEjC_8Jp1b3-sHKAg%40mail.gmail.com.

[Jonathan Arnett]

I am also a fan of the proposal. One addendum that I would like to suggest is adding a flag to IEx such that a library can be "installed" when the shell starts.

e.g.
iex --install :decimal

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/0100017725401617-8cdc5c5f-8434-42d6-a3b5-0272cbf65824-000000%40email.amazonses.com.

[José Valim]

> The latter solution could most easily be solved by using /tmp and making a new folder each iex instance that calls mix install (possibly also respecting the cache dir flag options)

Good points.

That's kind of how the current implementation works. The cache is in tmp and the cache key is the MD5 of the deps given to install. This means that the cache is most likely to be used only by the current script. So it reduces the odds of interference between scripts. Otherwise you will have to fetch and compile dependencies every time the script runs, which would be too slow.

Issues like Elixir version and Erlang version can be solved in two ways: 1. don't do anything because Mix already solves those or 2. add those to the cache key - either as part of the directory structure (my vote) or as part of the MD5 computation.

We could also support explicit naming for better control over the cache - but I would wait for those until people effectively run into cache issues, if ever. :)

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/62020d79-4f44-38d5-65e7-a786808da633%40gmail.com.

[Fernando Tapia Rico]

+1 to the proposal

+1 to add Elixir/Erlang versions as part of the cache dir structure

Another suggestion: I would explain in the documentation how to pass configuration to the dependencies :)

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4JFHs6stz5sHvFabZXP%2BS3HXMLN8PKd%2BVNJ%3DmwbF8-J_A%40mail.gmail.com.

[Boris Kuznetsov]

Recompilation (and delay) on every new run seems like a major issue.

Probably, if you call `Mix.install/?`, then you should pass the target directory to cache compiled dependencies.

By default it can use `_build` as all current mix projects.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4JFHs6stz5sHvFabZXP%2BS3HXMLN8PKd%2BVNJ%3DmwbF8-J_A%40mail.gmail.com.

[Wojtek Mach]

On 25 Jan 2021, at 11:40, Boris Kuznetsov <m...@achempion.com> wrote:

Recompilation (and delay) on every new run seems like a major issue.

Probably, if you call `Mix.install/?`, then you should pass the target directory to cache compiled dependencies.

By default it can use `_build` as all current mix projects.

The fake Mix project contains both deps/ and importantly _build folders so recompilations won’t happen once a given set of dependencies given to Mix.install is cached.

Recompilations would only happen if the dependencies change as is the case of `:path` dependencies. It’s actually pretty cool that `Mix.install [{:foo, path: “~/src/foo”}]` works out of the box and would recompile on start if that dep changed.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/1A4436D3-17AC-41BB-94CB-57F45D0ECF37%40achempion.com.