no-deps-check no longer respected in Mix 1.3.1?

[Eric Merritt]

Guys,

 
I am working on upgrading elixir support for nixos to Elixir 1.3. There seems to be a change in how Mix responds to the `--no-deps-check` flag. Previously, mix wouldn't try to do anything with deps. Now it blows up with an error when running the following.


```
   HEX_OFFLINE=1 mix compile --no-deps-check
```

The error that I see is:

```
* (Mix) Failed to open Hex registry (file does not exist, run `mix hex.info` to fetch it)
```

 Previosly
 This is new behavior between the previous version of Mix and the new version.  This problem essentially breaks elixir support on nixos, so I would love to get it fixed.

Eric

[José Valim]

Thank you Eric. I can confirm this is broken. Unfortunately we introduced this regression when fixing another bug. I will discuss a fix with Eric MJ and let you know.

If we introduce a new flag or an environment option, would that be acceptable? Since it will only be checked with future versions, it should not affect previous versions and effectively fix the issue for future ones.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

--
You received this message because you are subscribed to the Google Groups "elixir-lang-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-ta...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/e743d7dd-0808-467a-9963-754e611d22fe%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Eric Merritt]

Jose,

 That's perfectly acceptable. My main concern is that there is a way to tell mix that dependencies are already taken care of and it doesn't need to reach out to the network. How it goes about that, as long as its documented, isn't that big a deal. I will wait to upgrade until you guys release a fix.

Eric

Sent from ProtonMail, encrypted email based in Switzerland.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/CAGnRm4%2BjAdweauep-PXz7mfg1iqS%3Ds_RbOp-UeBi5Wdr_C0vjA%40mail.gmail.com.

[José Valim]

Eric, after a quick conversation with Eric MJ, he mentioned this should work with the same flags as parameters as long as you have a mix.lock file, as Hex no longer requires a registry for locked dependencies. I was able to verify such in a project like Plug. Which Hex version are you using? Have you tried the latest?

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/JyXKrpMQOisFenZyvqRjZZW1wAj2vve0IA8ivc9edcWChfDrndyOuR0-2pgGtrKLoWtAWmX-PFB6FeITi4ywMw%3D%3D%40merritt.tech.

[Eric Merritt]

Jose,

 I just upgraded the version of hex and am still seeing the problem. My suspicion is that some number of the elixir packages  in hex do not include a mix.lock. That said, the optimal course for me is that mix simply doesn't try to manage dependencies, regardless of whether there is a lock file or not. In the worst case, I can have the nix infrastructure generate a lock file.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/CAGnRm4%2Bo%2BPP6h6k1myBCGtpOu1b0%2BrDp6qDA_k0hLfed%2B%2BZC1Q%40mail.gmail.com.

[Eric Meadows-Jönsson]

The lockfile of dependencies are never used. So the lockfile of the top level project must be present.

Can you show a reproducing mix project where all dependencies are locked and hex still makes external request so that we can debug it?

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/dTVRCb_Dd-Mlbj4hwxMJCR0EtyG1TMAW-OgNvvho2lhUvxXm0PW-l1-2iAAj3h_5LHqKU5LlBVj7nxgI32v34w%3D%3D%40merritt.tech.

For more options, visit https://groups.google.com/d/optout.

--

Eric Meadows-Jönsson

[Eric Merritt]

I haven't seen that behavior, but its not related to my problem. I suspect some of the packages on hex probably don't have lock files. The thing I am wrestling with is the change in behavior from mix pre 1.3 and the current. Historically,  `no-deps-check` meant 'don't do any dependency checking`.  It did exactly the right thing, it assumed things where good and went and built. Now it seems to mean no dependency check as long as some dependency check has been done in the past, (ie the existence of a lock file). The original behaviour was much more useful to my (admittedly edge) case than the current behavior.

The information I need now is whether you guys consider this change in behavior a bug and are going to resurrect the original behavior or you consider it a feature and are going to leave it in place. In the first case I will wait until the fix is released, in the second case I will modify the nix infrastructure to generate the lock file.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/CAM_eapjotFeKg89iVQKDFE_RQCiwmKRvRLG3ChmBwHLknCgQ%3Dg%40mail.gmail.com.

[Eric Meadows-Jönsson]

> I suspect some of the packages on hex probably don't have lock files.

As I said, this is not an issue, since we don't use lockfiles of dependencies. In fact, we don't include lockfiles when publishing packages.

Can you explain the issue you are having? If you have a lock file you should not be seeing the error you had in your first post. Is the issue that you are still having this error or is that you don't have lock file? If it's the latter, how come you cannot use mix to generate the lockfile?

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/du1_yZ7gSQ5fe4i4udrIa_SqS3qffol4zrczzY8pdTJJIkWr8W-e_wQ7i0dqn5vVmSAfikpsX8wXzOH6m5sYaw%3D%3D%40merritt.tech.

For more options, visit https://groups.google.com/d/optout.

--

Eric Meadows-Jönsson

[José Valim]

> The information I need now is whether you guys consider this change in behavior a bug and are going to resurrect the original behavior or you consider it a feature and are going to leave it in place. In the first case I will wait until the fix is released, in the second case I will modify the nix infrastructure to generate the lock file. 

To give some background, we have introduced this backwards incompatible change because in previous versions Elixir would always add everything in your _build directory to the load path. Once we started sharing build paths for umbrella projects that became a problem. So now we need to at least now our dependency tree, so we know what are the important directories to load from _build.

What we need to decide collectively is:

1. Loading everything in _build is indeed error prone and we should not allow it. This probably calls for adding a mix.lock. I don't believe you can generate a mix.lock by hand as the lock formats can change any time. If we are choosing this option, I think the most sensible choice here is that Nix won't support packages without a mix.lock. If that's too common, then we probably should not choose this option.

2. Loading everything in _build is fine under some special circumstances as yours and we should add a flag that supports it

I am truly fine with both options, I would just like us to take an informed decision about it and having some examples of projects that currently fail will be a step in this direction.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/CAM_eaphhZJLxHzaOAgwSLsqyiNZ5ZPvN50ujrJR-KE_c83vBcw%40mail.gmail.com.

[Eric Meadows-Jönsson]

I don't know much about nixos so I don't really understand the use case and need to know more about how you are using packages to help you.

Do you use hex packages to build nixos packages? Hex packages and mix are of course not built to support this use case so there may be some compatibility issues.

Hex packages only include the necessary information to be used as a hex package, we don't include a lock file and to use it we assume that you have the hex registry available.

How come you are using hex packages instead of using the full project repositories from source control?

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/CAGnRm4JqjKRb12_qS_pdLijPvexey1oD62R35d-8ht_AVepXfQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Eric Meadows-Jönsson

[Eric Merritt]

Nix is a package management system designed to (among other things) reproducibly build artifacts, up to and including systems. What this means for build tools  is that the 'build' needs to be split into two parts; dependency resolution and compilation. Dependency resolution needs to be handled completely by nix, with compilation handled by whatever language specific tools are involved. 

My goal here is to get Mix to delegate dependency resolution to nix. We have done this for all the common beam platform tools. For some, (I am looking at you rebar) it was a massive pain for others (previous versions of mix and erlang.mk) it was very easy. That is the source of confusion. This was a non-backwards compatible change that seems like it was unintended, but may be kept around. I just need clarity on which it is so I can resolve the packaging problem it created for me. Obviously, I would prefer the old behavior where 'no deps check' meant 'just assume the deps are good and don't do anything deps related', but either way its resolvable. Its just a matter of the work involved.

As for why we based the package on Hex: Its so we could get 2000(ish) working packages into nix in one fell swoop. This is a very common paradigm in the nix world because its an easy way to get a large number of packages for very little cost. It generally works quite well, and has worked quite well for rebar3/mix/erlang.mk based beam projects as well. We wrote a tool that does translates Hex Packages to Nix Packages.  There is no practical way to do that automatically directly with Github. Of course, there is also nothing stopping people from manually creating a package that uses github.

That said, even if we did pull from Github, it wouldn't change the need to figure out a way to get mix to delegate dependency resolution to nix.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/CAM_eapi4jp8rVD-BmFoqifvmn%2BBxLj9wo%2BrE7GiHLSwe%3DA2eBg%40mail.gmail.com.

[Aaron Jensen]

I ran into the same issue with

* (Mix) Failed to open Hex registry (file does not exist, run `mix hex.info` to fetch it)

while trying to compile exsyslog in a docker container. I'm unable to narrow down a repro, everything else I try works.

This was my attempt to make a repro, but all I could repro was a hang while compiling syslog:

https://github.com/aaronjensen/mix_hang_repro

Should I report that somewhere?