Depending on a sub-application from another umbrella app?

[Myron Marston]

My team has an existing umbrella app. Another team at the company where I work is getting started building a new Elixir application and is starting it in another git repository. There area a number of sub-apps in our umbrella app that the other team would like to be able to use from their new application. They’ve tried a number of things but haven’t found a way to be able to depend on any of our sub-apps using a :git option. Ideally we’d like to be able to get this to work:

defp deps do
  {:foo, git: "g...@github.com:my_org/our_umbrella.git", path: "apps/foo"}
end

Is there a currently supported way to get this to work, or are we going to have to extract the bits they want to use into separate, stand-alone git repositories (something we’d rather not have to do)? Or is it something the Elixir team would consider supporting in a future version of Elixir? Is there a suggested work around for now?

One idea I had is to run a private hex server and publish our sub-applications to that…but I have no idea how to run a private hex server or how to depend on a package from a private hex server (and my few minutes of searching hasn’t turned anything up).

Thanks!
Myron

​

[José Valim]

Ideally we’d like to be able to get this to work:

defp deps do
  {:foo, git: "g...@github.com:my_org/our_umbrella.git", path: "apps/foo"}
end

Is there a way to checkout only a subdirectory in Git? It seems to be possible, so someone could contribute this feature to the Git SCM. Although I would call the option something other than :path.

Is there a currently supported way to get this to work, or are we going to have to extract the bits they want to use into separate, stand-alone git repositories (something we’d rather not have to do)? Or is it something the Elixir team would consider supporting in a future version of Elixir? Is there a suggested work around for now?

The only option for now is to break those into git dependencies. We are working on private Hex servers and registries, so that should be an option soon. One of these two options are arguably the best approach: since those applications are now used by different projects, it probably makes sense to move them elsewhere.

[Myron Marston]

Is there a way to checkout only a subdirectory in Git? It seems to be possible, so someone could contribute this feature to the Git SCM. Although I would call the option something other than :path.

 It appears that this kind of checkout is possible in Git 1.7+ using a feature called sparse checkout.  I'm not sure that that alone would completely solve the issue, though; I had hacked together an attempted work around in the other team's app like this:

# in root mix.exs file
defmodule TheirUmbrella.Mixfile do
  use Mix.Project

  def project do
    [apps_path: "apps",
     build_embedded: Mix.env == :prod,
     start_permanent: Mix.env == :prod,
     aliases: aliases,
     deps: deps]
  end

  defp deps do
    []
  end

  defp aliases do
    ["deps.get": [&fetch_our_umbrella/1, "deps.get"]]
  end

  @sha "3d237160a52aa38a33a8413e54e51ad3bf3b041a"
  @repo_url "g...@github.com:our_org/our_umbrella.git"

  defp fetch_our_umbrella(_args) do
    if File.dir?("our_umbrella") do
      File.cd!("our_umbrella", fn -> System.cmd("git", ["fetch"]) end)
    else
      System.cmd("git", ["clone", @repo_url, "our_umbrella"])
    end

    File.cd!("our_umbrella", fn -> System.cmd("git", ["checkout", @sha]) end)
  end
end

​

And then in one of the sub-apps in their project we added {:foo, path: Path.expand("../../our_umbrella/foo", __DIR__)}. This worked since foo was a simple dependency that had no :in_umbrella dependencies of its own. But once we tried to depend upon one of our sub-apps that had an :in_umbrella dependency, it failed, because mix did not understand that it was an umbrella dependency from our umbrella instead of theirs.

So I think that cloning just a subdirectory out of a git repository isn’t quite sufficient, and in fact it might not be the best approach since you may need other directories for :in_umbrella dependencies.

That said, I recognize that this is pretty complex and is probably going far beyond what umbrella apps were designed to do :).

The only option for now is to break those into git dependencies. We are working on private Hex servers and registries, so that should be an option soon. One of these two options are arguably the best approach: since those applications are now used by different projects, it probably makes sense to move them elsewhere.

From a purist’s point of view, it absolutely makes sense to move these into separate git repositories. But from a pragmatic point of view, that is undesirable for us. Since we are the primary creators, users and maintainers of these libraries, it has been really helpful to have them in the same git repository. It means that our CI builds uncover integration issues between a dependency and its consumer within our umbrella, and that it is easy for us to change a library and the consumer in the same commit—kinda like how for elixir-lang development, you’ve found it useful to keep mix, ex_unit, iex, etc in the same repository with elixir itself. It facilitates simpler development workflows.

Even if we wanted to split these into separate git repositories, we’d probably have to recon with the :in_umbrella issue I mentioned above: if the other team wanted to depend upon just one of our apps foo, if foo depended upon bar and baz in our umbrella, it would not be sufficient to just move foo into a separate git repository (since it would run into the very issue the other team has run into). We’d also have to move bar and baz out into their own repositories so that foo could depend on them. Suddenly my team has to maintain code across 4 git repositories instead of 1. That’s definitely undesirable.

It sounds like running our own private hex server might be the right approach, unless you want to mix dependencies to handle the :in_umbrella issue when pulling in from another umbrella. Any idea when private hex servers might become viable?

Thanks,

Myron

​

[José Valim]

> But once we tried to depend upon one of our sub-apps that had an :in_umbrella dependency, it failed, because mix did not understand that it was an umbrella dependency from our umbrella instead of theirs.

That's probably solvable though. Do you remember which error in particular did you get?

> It sounds like running our own private hex server might be the right approach, unless you want to mix dependencies to handle the :in_umbrella issue when pulling in from another umbrella. Any idea when private hex servers might become viable?

Eric should have an idea but no guarantees.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

[Myron Marston]

That's probably solvable though. Do you remember which error in particular did you get?

Here’s what I get:

➜  shaardwolf git:(myron/use-delorean) ✗ mix deps.get
Dependencies have diverged:
* util (/Users/myron/moz/shaardwolf/delorean/apps/util)
  the dependency util in apps/crawl_processor/mix.exs is overriding a child dependency:

  > In apps/crawl_processor/mix.exs:
    {:util, nil, [path: "/Users/myron/moz/shaardwolf/delorean/apps/util", manager: :mix]}

  > In delorean/apps/shard/mix.exs:
    {:util, nil, [path: "../util", in_umbrella: true]}

  Ensure they match or specify one of the above in your deps and set "override: true"
** (Mix) Can't continue due to errors on dependencies

To explain what the various named things in the error are:

• delorean is my team’s umbrella app
• shaardwolf is the other’s team’s new umbrella app
• shard is a library application in delorean at apps/shard which has an in-umbrella dependency on util (which is defined at apps/util in delorean)
• crawl_processing is a sub-app in shaardwolf which is declaring a dependency like so:

defp deps do
  [
    {:shard, path: Path.expand("../../delorean/apps/shard", __DIR__)}
  ]
end

Delorean is cloned in delorean (under the shaardwolf root) via the snippet I pasted before.

One work around I had in mind was to change our dependency declarations in delorean to use path: Path.expand(...) instead of in_umbrella: true. I tried this on the delorean side, but it triggers errors there:

➜  delorean git:(myron/use-path-instead-of-in-umbrella) ✗ mix deps.get
Dependencies have diverged:
* util (apps/util)
  the dependency util in mix.exs is overriding a child dependency:

  > In mix.exs:
    {:util, nil, [path: "apps/util", from_umbrella: true, manager: :mix]}

  > In apps/admin/mix.exs:
    {:util, nil, [path: "/Users/myron/moz/delorean/apps/util"]}

  Ensure they match or specify one of the above in your deps and set "override: true"
# ... followed by a similar message for every single in-umbrella dependency

I’ve come up with a work-around for this that actually seems to work, though — in delorean, I’m conditionally defining the in-umbrella dependencies using either {:in_umbrella, true} if System.cwd indicates we are in delorean or a subdirectory (meaning we are developing delorean) or {:path, Path.expand("../apps/#{dep}", __DIR__)} otherwise (meaning delorean has been cloned into shaardwolf and that is being developed). It’s quite the hack but I’ve confirmed it works OK on both sides!

Myron

​

[José Valim]

Given your report, I cannot understand why it does not work. We consider path deps equal if they point to the same destination and in all cases (umbrella sibling, umbrella children and regular path deps), the destination is expanded, which means they should point to the same place:

https://github.com/elixir-lang/elixir/blob/master/lib/mix/lib/mix/scm/path.ex#L22

If you can provide a way to reproduce the error using my umbrella repo (https://github.com/josevalim/umbrella_sample) or another repo, I can take a look. :)

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CADUxQmtx7PzMoYwjZxHZcN%3DJRyqQACL%2BOXJr%3Dg%2Bhs_irB20fvA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Myron Marston]

Thank you, José!  I put together a repro here:

https://github.com/josevalim/umbrella_sample/pull/5

Turns out, there was something I was doing that I didn’t realize caused the issue:

defp deps do
  [
    {:util, path: Path.expand("../../delorean/apps/util", __DIR__)},
    {:shard, path: Path.expand("../../delorean/apps/shard", __DIR__)},
  ]
end

The fact that I was declaring a dependency on both util and shard (and not just on shard, as I had said before) caused the error.

Myron

​

--
You received this message because you are subscribed to a topic in the Google Groups "elixir-lang-core" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elixir-lang-core/aynMv_jMNFw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4JiakRM5FcPxoaDJ%3D01vEO6_XHKFFHfAoENy%3DeoTdTppA%40mail.gmail.com.

[Eric Meadows-Jönsson]

> It sounds like running our own private hex server might be the right approach, unless you want to mix dependencies to handle the :in_umbrella issue when pulling in from another umbrella. Any idea when private hex servers might become viable?

To clarify, there are no plans for private hex servers under the hexpm organization. We have the hex_web server that is primarily meant to host hex.pm and is probably too much for most private hex servers (there are lots of things you wouldn't need), but I would still accept patches that make running hex_web yourself easier.

Running a private repository is quite easy though; you just need to host the package tarballs and the registry file on HTTP endpoints and that's it, you can build tarballs with `mix hex.build` and the registry format is specified. If someone builds software that makes some of these parts easier we could to add it to the hexpm organization on github.

You can run your own hex repository today and the Hex client supports it (just set the HEX_REPO environment variable). What is planned is to make using packages from multiple repositories easier so you can pull packages from hex.pm and your private repositories at the same time. The plans for how this should work is all specced out and a few people have shown interest in working on it so I expect this to be supported soonish.

On Tue, May 17, 2016 at 6:14 PM, Myron Marston <myron....@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/61360ece-4725-433f-a684-6fccca64a2f4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Eric Meadows-Jönsson

[José Valim]

Thank you, José!  I put together a repro here:

https://github.com/josevalim/umbrella_sample/pull/5

Thank you, I have found and fixed it.

All dependencies run in a particular environment, which defaults to production. The conflict you were getting was not because of the path but because the dependencies were configured to run in different environments. In order to make it easier to catch such errors, we will also print the :env value when printing dependencies.

However, it was not really different environments, one had env: :prod and another one had no :env, so when they compared the failed but they should not, because the default :env *is* prod. So I have made sure we set the default env to prod as well, effectively solving the error.

See commit: https://github.com/elixir-lang/elixir/commit/eced1303efd2622f5080949290bdd9dc9b92c893