Proposal: Mix Deps equivalent of Bundler's Local Git Repos

[Chris Keele]

While I love the local git repo feature of bundler, maintaining the invisible bundler configuration and having both local and global options is awful.

I noticed that mix doesn't have this ability and I thought maybe we could get it right.

Currently, there are two mix dep sources that don't go through hex, git and path sources.

The goal would be to come up with a way of specifying a dependency such that mix will use a path source to a dep under certain toggleable conditions, but the git source by default. Additionally, it would be great to somehow avoid having the actual local path source provided appear in the mix.exs config. This requires some sort of hidden state, obviously, but lets package developers transition smoothly between developing and testing packages without risking bad commits or deploys.

My thoughts were to perhaps have a new key in mix.exs project configs, similar to other environmental ones:

def project do
 [build_embedded: Mix.env == :prod,
 use_local_deps: Mix.env == :dev,
 deps: deps]
end

defp deps do
 [{:mydep, git: "https://github.com/elixir-lang/mydep.git", tag: "0.1.0"}]
end

Then notice the following sort of application configuration in some config/local.exs type file that could be git ignored or avoided being loaded at developer discretion:

config :mix, local: [

 mydep: "~/Projects/oss/elixir/mydep" 

]

When use_local_deps is true then every dep found in the :mix, :local configuration would act like a path source instead of a git one, provided that a tag or branch is set in the git source and is currently true of the local path.

Do we want to support this feature? Is this the best way to configure it? I don't know. I've just started to get tired of toggling alternate commented-out path and git versions of deps, and either accidentally deploying with a path dep, or forgetting to properly update a git dep that I'm also developing locally and pushing frequently.

Depending on how people feel about this feature and its implementation syntax I'd love to work towards a PR for this.

[José Valim]

Hey, I was the one who implemented that feature in Bundler. :)

In your particular case, have you tried using deps/your_dependency as a git repository? One other possible way of supporting this is by using environment variables and handling the logic in your mix.exs.

{:foo, path_or_git("foo", "company/foo")}

where:

defp path_or_git(path, git) do

  if System.get_env("USE_PATH") do

    [path: path]

  else

    [github: git]

end

In particular, path dependencies were written so they never change the lock, so it already provides part of the functionality I had to implement for the local git repo thing. I am not sure if you won't run into other issues though but it is worth exploring and knowing what those issues would be.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/bb2391f9-5471-4e48-a4d2-91b32ffa7edc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chris Keele]

Hey, I was the one who implemented that feature in Bundler. :)

Hah, that's cool. I swear you've written half the software I really like using. :)

  

One other possible way of supporting this is by using environment variables and handling the logic in your mix.exs.

 

My only complaint with bundler's way is rooted in its non-explicit and spread-out configuration system. It takes foreknowledge and several commands to set up local repos or figure out what's going on, so I like the idea of something more explicit and centralized than ENV vars.

In particular, path dependencies were written so they never change the lock, so it already provides part of the functionality I had to implement for the local git repo thing.

That's great, I didn't know that! This is really the only reason why this would have *needed* to be a Mix feature request instead of a useful mix.exs pattern.

I'll play around with in-mix-file options and see if/how they work best, and report back if there's something worth adding to mix.

[Chris Keele]

Here's a dirty implementation within a mix.exs: https://gist.github.com/christhekeele/5bbc3ad76bf194adda3f5a822b60b4a4

Research shows that:

• This totally works, try it with/without MIX_ENV=prod
• It even works for overrides of implicit deps
• Interestingly if both git and path sources are both specified, git is preferred
• My config idea obviously wouldn't work; config.exs isn't available when mix.exs is loaded

The reasons why it might make sense to implement this internally in mix anyways:

• Promote this pattern
• Bake the branch/tag/ref requirement into the dep fetching process so that nothing is fetched/compiled if the requirement isn't met, at the same point in its process that emits the dependency does not match the requirement "1.0.0", got "1.1.0" (instead of halfway through the compile if there's a version mismatch between the dep version requirement and the locally checked out commit)
• Actually check the local git repo for if it satisfies the git requirements (branch/tag/ref) at that point
• Much easier to future-proof this: Mix.Deps structs know if they use any scm, my code has to check each permutation of (git, github|branch, tag, ref) and assumes no new options will be added
• Might allow us to find a way to specify the local deps outside of the checked-in mix.exs file without requiring it manually load an external file that might not exist during deploy (assuming the file listing the deps is gitignored), seems a little iffy

[José Valim]

Great job!

Given Git has higher priority, maybe we could support a :local option (or something of sorts) inside Mix.SCM.Git that will behave exactly as you describe.

Alternatively, you could implement some sort of LocalGit SCM where it looks for one key in particular (like Git and Path do). Being a separate SCM could be useful as it could be brought in as a third-party dependency (an archive in this case).

I definitely recommend you to further explore those solutions and see which one makes the most sense. :)

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

[Chris Keele]

I've been looking at this and playing with some code. I've realized I've brought in some assumptions from bundler that I've never questioned.

Namely, how strict should we be with the aligning git repo state and local dependency override?

The key feature, to me, is having the ability to override a remote dependency with a local version, for development's sake. I'd never thought before about why that might be restricted to git development. I see the case for some sort of 'integrity' benefit from aligning the local version with a specific git ref (branch/tag/ref) in case the mix.exs gets committed. But the only time I've ever noticed this requirement is when I've made a new branch in my local repo for perfectly legitimate development workflow reasons and suddenly I've broken bundler. On the other side of the coin, this requirement does nothing to actually ensure that the commit you have to ref out locally is actually available on the remote, so any assurance of integrity seems a little suspect to me. I break my workflow all the time this way, too. So questions:

1. Should we care if the local git repo isn't at the specified ref (branch/tag/ref) in the mix.exs?

2. Should we care if there isn't even a ref in the mix.exs?

3. Should we care if the local path isn't even a git repo? What if that dep somehow uses mercurial for development?

4. Should we even care if the remote dependency we're overriding with a local version isn't a git one? Why not override a remote hex dep temporarily for local dev?

The more we loosen these restrictions, the more powerful a local path config would become, the simpler the code would be, and fewer potentially invalid mix.exs states would be available. On the other hand, one could argue that without these checks you could shoot yourself in the foot––but it's still an improvement on the `committing path dependencies when you don't mean to` situation and I'm no longer 100% convinced they actually protect against common errors so much as limit your overriding workflow.

I'm no longer sure how strict we should be. It's super handy that you implemented this for bundler because I imagine you've already discussed the pros/cons of these restrictions with the bundler team and have some thoughts on them.

[Chris Keele]

(Worth noting that #4 would actually require a non-trivial restructuring of Mix Deps. The others, however, are trivial.)

[Chris Keele]

Also worth noting that I'm planning on attempting the restructuring anyways just to see how it goes–I feel as if the loader, converter, and fetcher would benefit from a more unified Mix.Dep.Source struct that handled local/remote, git/path/hex, rebar/archive stuff with the same interface. But that's for later.

[John W Higgins]

This seems so much more complex then needed.

I would also question why path is the be all and end all of the conversation. I could just as easily have different git repos for a dependency based on environment.

So why not a dead simple change to the way we can assign dependencies and have fun.

For example - if one were to add an ability to declare deps as a map (which could easily be reduced back to the exact same format internally so it's as small a change as possible) - we would be able to do something like this

  def project do
    [
     deps: Map.merge(deps, deps_env)
    ]
  end

  defp deps do
    %{
      dep_x:, {"~> 2.1"}
    }
  end

  defp deps_env do
    case Mix.env do
      :prod -> %{}
      :test -> %{}
      :dev  -> %{
                  dep_x: {git: "https:/xxx", tag: "3.0-dev"},
                  dep_y: {"-> 1.0"}
                }
    end
  end

The merge would override dep_x with the local version in dev (or test or prod if you wanted) with the exact same format as you would normally declare it with (assuming the map option).

Doesn't that do everything you would want? More importantly it doesn't set any "rules" on anything at all - it just allows for very nicely laid out options.

John

[Chris Keele]

Hey John!

Your example doesn't the quite do the major thing I need it to: stay out of the mix.exs file.

Think of it like shadowing a variable, but for dependencies. My goal here is to be able to temporarily override a dependency with a local path, to test drive the dependency's development, in a way that I know will never accidentally get committed and break the host project's build. It has to be a way that is git-ignored so people I collaborate with can use their own local shadowings without merge conflicting over mine. I go into a little more depth on how and why bundler does it and how we could as well on the scratchpad PR I'm using to experiment with this feature.

The gist I originally linked to wasn't meant to emphasize that this was possible within the mix.exs, that's a simpler task. It was to emphasize that using overrides is compatible with the mix.lock and mix deps system at large, so we could work on finding a way to remove it from the mix.exs. But not having it be in the mix.exs is really the core requirement––which is exactly why I'm considering stripping away the other restrictions used in the bundler system; I agree with you that these rules are fairly cumbersome.

You're also right that paths need not be the end-all-be-all, with a general override system, but right now that isn't quite so simple because hex packages exist at a different level of abstraction within mix than git and path packages. And it doesn't make much sense to override a path dependency with a git one. That leaves just the path-overriding-git vector, until I work on the refactor mentioned earlier to bring them to the same level of abstraction.

[Chris Keele]

There are definitely outside-of-Mix ways to meet these needs, for example:

def project do

  [deps: build_deps]

end

defp deps do

  [foo: "~> 1.1.0"]

end

defp build_deps do

  if user = System.get_env("MIX_USER") do

    Keyword.merge deps, apply(User.Deps, user, [])

  else

    deps

  end

end

defmodule User.Deps do

  def keele do

    [foo: [path: "/home/keele/projects/foo"]]

  end

  def john do

    [foo: [github: "wishdev/foo"]]

  end

end

But every way I can think of pollutes your mix.exs, git history, and environment with what is strictly local, your-development-machine-only configuration. Building a little something in to Mix to help do this keep everyone's builds working and configuration clean. But I agree many other forms of dependency business logic, like Mix.env driven ones, can and should be right alongside everything else in your mix.exs so you know what's going on and can stay as flexible as possible.

[Jordan Day]

Rebar3 has the concept of "Checkout Dependencies" (https://www.rebar3.org/docs/dependencies#checkout-dependencies) in which the dependency is specified normally in your rebar.config (git repo, hex package, etc.), but *if* you create a copy of that project in a _checkouts directory, that copy winds up being used.

So, it's not exactly flexible per-environment, but it's quite useful for what I would assume are two very common use cases: Working on a local copy of a dependency for debugging, or temporarily using a "fixed" version of a dependency while waiting for the "official" version to get updated in hex/PR merged in the repo/etc.

I personally like the idea of "checkout dependencies" because it's very simple and doesn't try to do too much, while covering what I would assume to be the most common reasons for wanting local copies of a dependency.