Move Dependency Build files into a seperate location

[l...@expert360.com]

The _build folder contains all build artifacts for external dependencies and internal application code. This means that caching this folder is a pain.

If I could simply cache the external dependency build artifacts separately to the internal application artifacts, this would solve my problem. To do this, I think the artifacts should be kept in a seperate location such as _build/<env>/deps/...

[Paul Schoenfelder]

Could you clarify what the issue is? You haven't specified under what circumstances caching is a problem or with what tools.

Paul 

On Mon, Dec 18, 2017 at 10:53 PM <l...@expert360.com> wrote:

The _build folder contains all build artifacts for external dependencies and internal application code. This means that caching this folder is a pain.

If I could simply cache the external dependency build artifacts separately to the internal application artifacts, this would solve my problem. To do this, I think the artifacts should be kept in a seperate location such as _build/<env>/deps/...

Want to get insights, recommendations and answers from top consultants for free? Head to Discussions to ask questions and find answers.

———————————————————————————————————————————————

This email is intended for the sole use of the intended recipient(s) and may contain information that is confidential and/or privileged.  Any review or distribution by any other person is prohibited.  If you are not an intended recipient, please immediately contact the sender and delete all copies of this email message

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/ac3428e7-f626-4392-ab3a-9aa32a23face%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Louis Pilfold]

Hi gang

There's a mix task that will remove compiled application code (clean, I think). Run this before caching the build directory :)

Cheers,
Louis

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAK%3D%2B-Tsp3Q6HbzrE1kJ0Th1GUAxreZyVckU7d4kFY_8JfmAoCg%40mail.gmail.com.

[l...@expert360.com]

Ah, so to give some context:

We have a CI pipeline which pulls the deps, builds the umbrella application and then runs our tests.

I would like to cache pulling and buildings the deps, and just build the umbrella application. It would save some time off the CI build. We use mix to do our compilation, testing and every other command.

[Michał Muskała]

A solution for this could be to do something like:

export MIX_ENV=test
mix deps.get
mix deps.compile
cache_build_artifacts
mix compile
mix test

This should make it possible to cache the dependencies separately from
the application.

> https://groups.google.com/d/msgid/elixir-lang-core/6d6eabdc-ee00-4a0e-8da3-4762358b8683%40googlegroups.com.

[Paul Schoenfelder]

My approach, at least under Docker, is more or less what Michał showed - use mix deps.get, deps.compile in one step, cache the results, then do everything else on top of the cached layer. Your situation should be more or less identical if I'm understanding you correctly, so I believe that is all you need here.

Paul

> email to elixir-lang-core+unsubscribe@googlegroups.com.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elixir-lang-core/6d6eabdc-ee00-4a0e-8da3-4762358b8683%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGAFNpkAbD3cGZNnnkERPeBY8%3DXg6FsuLAny%3D8nxbA%2BiM1ktRQ%40mail.gmail.com.

[l...@expert360.com]

The deps.compile within an umbrella application actually compiles some of the internal applications. I would actually need to do:

mix deps.get 

mix deps.compile

mix clean

cache_build_artifacts

mix compile

mix test

This is another reason to move the build artifacts to another location for deps.

[Paul Schoenfelder]

That happens because in an umbrella some of your deps are applications in the umbrella. Separating the build artifacts would still compile the umbrella applications which are in the dependency tree.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/147180e8-d542-4257-b059-90ab3c2f86bd%40googlegroups.com.

[José Valim]

The deps.compile within an umbrella application actually compiles some of the internal applications. I would actually need to do:

You can still use a Mix task to get the name of the umbrella applications and figure out what you want to cache or not - or even have the mix task doing the caching. It seems clear only relying on the file system is not enough for you but I think you should focus on caching more programmatically rather than expecting Mix to lay out the perfect directory structure for you.

We need a very strong reason to change the _build structure which has been settled from before Elixir v1.0.

--

José Valim

www.plataformatec.com.br

Founder and 

Director of R&D

[Wojtek Mach]

> The _build folder contains all build artifacts for external dependencies and internal application code. This means that caching this folder is a pain.

Could you elaborate why is this painful? Is it the sheer size of the _build directory? Or you don't want to cache app code? Please note that e.g. heroku-buildpack-elixir caches the whole _build directory, however uses `mix compile --force` to compile the project - this makes sure that your app code is always recompiled (works in umbrella too) but the deps are only recompiled when they actually change.

[l...@expert360.com]

I like your sentiment here and I have been debating doing this. I think I am just used to something like golang or javascript where the dependencies can be precompiled without necessarily being tied to one specific project and then passed to multiple build steps.

Being able to compile a dep for a specific version (of elixir, architecture etc) and then pass that compiled binary around is really what I am looking for.  

[José Valim]

That’s not really possible generally speaking because your project may have configs that change how a dependency is compiled. The cache therefore should be per project, which brings us back to _build and having a mix task that does it for you.