Proposal: Let Mix use dependencies precompiled by external build systems

[Alec Hartung]

Hello,

When I was optimizing our Nix-based CI, a pain point was having to transform dependencies from the Nix store representation into what Mix wanted.
This involved faking the _build/<env>/lib/<app> directory structure and Mix metadata, which was a lot of trial and error to prevent Mix from accidentally recompiling something that was just compiled.
I built mix_precompiled_deps to help work around that by configuring Mix to consume them from the Nix store directly.
We've been using it for a few months and it's been working really well.

This has enabled some neat things:
- `mix test` and similar inside Nix flake checks, without any additional compilation
- very optimized CI checks, where only dependencies that changed are recompiled
- local development with compiled dependencies coming from a shared binary cache
- on top of that, dialyzer PLTs coming from the shared binary cache too

It does require a few Elixir patches, though:
1. trust :build in Mix.Dep.load_paths/1
2. trust :build in Mix.AppLoader.load_apps/5
3. add a :precompiled opt to dependencies

All patches were written against Elixir v1.20 and apply cleanly against main.

The first two patches are to trust the paths that a Mix.SCM implementation provides, rather than enforcing the current directory layout.
This doesn't change behavior for the current SCMs and the rest of Mix already reads :build directly.

The third patch gives an option to skip compilation, when they're already precompiled.
Right now Mix recompiles all local dependencies, so this allows an SCM to mark a dependency as immutable and opt out of that.
I went with a per-dependency option here, but something like a new Mix.SCM callback could work too.

If there's interest, I can add tests to these patches and make pull requests.

Thanks!

Disclaimer: the patches were written with AI assistance and reviewed by me.

[more_than_3_characters]

Not the same exactly but I would add that it seems there isn't a way to keep hex tarballs & dependencies totally local in a predefined folder. When I'm running docker builds, even though I have the deps locally it seems it always needs internet. I think that there should be a way of doing something like `mix deps.get --local ./hex_local` in a format (tarballs, indexes, etc) that you can just copy into a folder (to the application folder you want for instance) and then when running in the context of that app being able to do `mix deps.get --only-local ./hex_local` where this only uses the contents of that folder. If any dep is specified in the mix.exs but is not available it would error out. Doesn't need to be with `mix deps.get` task, could be a new one `local_deps.get` or something else. Specially with the recent focus on CVEs, new AI risks, supply chain hardening and so on.

This would allow to do tests on the output of the compilation of the deps in a straightforward way that can be included in CIs as well using cached artefacts, external registries, etc.

Just my 2¢