[Proposal] Don't update git dependency when only origing is changed

[Rudolf Manusadzhian]

Hi there! In the projects I'm working on some dependencies are fetched directly from private git. First time latest master is fetched and then its sha is locked in mix.lock. No tags, no branches are used (sure, might be not the best practice). Lately I was moving "private" libraries from GitHub to GitLab and updating their sources in mix.exs. So they continue build in CI as before. However, when I ran mix deps.get - lock_status of private dependencies' evaluates to :outdated and deps get updated to latest master. Though the sha of previous revision is also available in new origin. 

(Currently I manually reverted parts of mix.lock file to point to the "previous" revision.)

Is this a valid use case to not update dependency but only update the uri of origin in mix.lock file?

Or maybe introduce new task like mix deps.update_origin <dependency_name>?

[Kenny Evitt]

The behavior you described seems like what one would generally want to happen.

You wrote:

> Though the sha of previous revision is also available in new origin.

So, in your scenario, BOTH the origin and the commit history of the Git repo changed. If you don't explicitly specify a ref/branch/tag in the dependency in your `mix.exs` file, then (AFAICT, I couldn't find explicit info in the Mix docs) the default is to use the `HEAD` of the `master` branch.

It's not obvious that Mix _will_ update a Git dependency just because the URL changed. That seems like a minor 'cost' regardless – changing repo hosts probably isn't very frequent for almost anyone.

If one of your repos is still accessible from GitHub, trying keeping the ref/branch/tag option for the "previous revision" you want to use, but switch the URL back to the GitHub version. Then run `mix deps.get` and that should test the behavior when ONLY the 'origin' (URL) changes. I suspect that Mix will consider the dependency to be up-to-date if the referenced commit is available.

[Rudolf Manusadzhian]

> So, in your scenario, BOTH the origin and the commit history of the Git repo changed.

I might not describe it clearly enough.. But commit history is the same. Only origin has changed. 

> Then run `mix deps.get` and that should test the behavior when ONLY the 'origin' (URL) changes. I suspect that Mix will consider the dependency to be up-to-date if the referenced commit is available.

That was my expectation as well.. but it doesn't work like that. It updates to the latest rev.. I did that with multiple repos with elixir 1.8.2 and elixir 1.11.4

I tracked it down to this part of Mix.SCM.Git.lock_status/1 where get_lock_rev/2 returns nil because first condition in if-expression (repo == opts[:git]) is not satisfied because repo here is new origin and opts[:git] is old origin. And as a result it always fails into the final condition `true -> :outadated`

I see in docs for Mix.SCM.lock_status/1callback it says:

> Note the lock may also belong to another SCM and as such, an
> structural check is required. A structural mismatch should always
> return `:outdated`.

 

But I don't think that's the case.

[Kenny Evitt]

Hmm – I'm definitely confused because you mentioned "the sha of previous revision is also available in new origin" in your first message.

And then you mentioned "It updates to the latest rev." in your last message.

Can you share the portion of the `deps/0` function for this dependency from your `mix.exs` file? (You can and should redact the URL and change the name if either is sensitive.)

Assuming there's no difference in the commit history between the two remote repos, it's a good question whether changing the remote repo URL is a "structural mismatch". I can sympathize with the possible decision by the Mix folk to just re-fetch the dep in this case tho.

And, worst case, it still seems like this would just be a one time 'cost', i.e. the dependency will be re-downloaded after changing the repo URL, but just once (until the repo itself, or the portion referenced in the dependency info, is updated).

[Austin Ziegler]

As far as I can tell, what’s being said here is that commit A is
available on GitHub. They moved to Gitlab and have done further
development, and the current HEAD is commit B. The process of moving
from GitHub to Gitlab did not cause commit A to fall off.

When the repo URL is changed from github.com to gitlab.com, Mix
detects this and refetches the current HEAD for the target branch,
getting commit B. But commit A is still available on gitlab, so it
feels like this update shouldn’t be necessary.

I suspect that there’s no good fix for this other than tagging commit
A and specifying that tag as part of the dependency.

-a

> --
> You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/5172d1a7-3871-481e-be12-b31f2d7bc5ddn%40googlegroups.com.



--
Austin Ziegler • halos...@gmail.com • aus...@halostatue.ca
http://www.halostatue.ca/ • http://twitter.com/halostatue

[Christopher Keele]

I would argue that today's behaviour is correct.

Conceptually, a dependency's version is only part of its identity; its source is the other component. When either changes, mix should re-install.

I'll bring up a few examples below. Note that I bring them up less because they'd be common in the real world, and more as evidence that we must think of the git source repository as a component of the dependency's identity.

1. Non-git dependencies re-install when sources change
   The default 'source' of a package is the hex.pm organization. When you change the :organization key of a remote dependency, or the :path key of a local dependency, you have to re-install.
   Changing the source of a :git dependency should do the same.
2. Git dependencies don't recognize semantically equivalent git refs
   That is, if a tag or a branch name points to the same commit, changing the :tag of that dependency in mix.exs to a semantically equivalent tag/branch/commit name triggers a dependency re-sync.
   This demonstrates that mix does not try to understand git semantics; a principle which, when uniformly applied, means our behaviour when the origin changes makes sense too.
3. Git shas are not unique between origins
   We often treat git commit ref shas like UUIDs, but it's important to remember they are hashes, and still have collision risks.
   A totally different origin with different source code might contain the same commit, and so mix should assume it may need to re-install if the commit sha is the same.
   This is more evident when you remember that a branch name like main or master may be what the :tag references, and we'd naturally assume that'd be different on different origins.

  

[Christopher Keele]

> Is this a valid use case to not update dependency but only update the uri of origin in mix.lock file?

More generally, I'm curious as to why you find this behaviour problematic—surely it's a trivial one-time cost for each location your project runs mix deps.get? Or are you changing git dependency origins often?

[Austin Ziegler]

I agree with the reasoning. I think that the correct thing for the
original poster to do is to not depend on hashrefs (which, as you say,
are not guaranteed to be unique across remotes) but on labelled tags.
While they are still subject to change (git tag push --force), they
are less likely to vary wildly between remotes of the same repo. The
other option (not available for all git dependencies) is to use a
combination of git and version (it is legal to say `{:dep, "~> 1.1.1",
git: "repourl", branch: "main"}`, but only if you set your version
usefully between releases. This isn’t something that all repos do
(there’s one dependency that I use which…doesn’t because the version
is set by the automated release process).

-a

> To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/ae7af5d0-aa13-44d7-a574-f386d377f7b3n%40googlegroups.com.

[Rudolf Manusadzhian]

Thanks all for the response.

> More generally, I'm curious as to why you find this behaviour problematic—surely it's a trivial one-time cost for each location your project runs mix deps.get? Or are you changing git dependency origins often?

I just faced this issue migrating from GitHub to GitLab around 10 elixir projects all of them had dependencies which are also moved to GitLab. I totally agree that tagging versions only come with benefits but those projects where already set to fetch from "master" the sea that's in lock file. Again, I agree that it's not ideal, just thought to share the case I faced and discuss the idea.

Thank you for the discussion.

[Christopher Keele]

> I think that the correct thing for the original poster to do is to not depend on hashrefs (which, as you say, are not guaranteed to be unique across remotes) but on labelled tags.

Worth pointing out that this does not resolve the OP's pain point: using labelled tags instead of hash refs does not prevent dependency refetching when changing the repo.

But definitely important to call out best practices in this regard!

You received this message because you are subscribed to a topic in the Google Groups "elixir-lang-core" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elixir-lang-core/vFVQ367Yeic/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAJ4ekQv0ZYp2ZY9-UoaM_%2BEHaJ4yassTxt9Vc8hBnGerP4x9xg%40mail.gmail.com.

[Austin Ziegler]

Mmm. It might not prevent the refetch, but it will (should) resolve
the issue of the locked ref updating because of refetching from the
main branch from the first message. I suspect that the network traffic
involved in the refetch is annoying, but the pain point is that the
dependency state changed even though it did not need to do so.

-a

On Mon, May 10, 2021 at 4:08 PM Christopher Keele

> To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAD9kT2TUHqFaL5QPx_o5PvPg6-yX3JtsOLYEdk4Mb91GUZW8Nw%40mail.gmail.com.