Fetching deps from mix.lock alone

425 views
Skip to first unread message

Miguel Palhas

unread,
Jun 12, 2018, 5:06:46 AM6/12/18
to elixir-lang-core
Hi

I'd like to know if there's any plans to add, or reasons against, having a way of fetching deps by using the mix.lock file alone, and not the mix.exs.
This would be similar to Ruby's `bundle install --deployment` I believe

The use case I'm thinking about, and currently suffering a bit with, is about building umbrella apps inside docker images.

The naive way of building a docker image would involve the following steps:

# copy everything to the container
COPY . .
# fetch deps
RUN mix deps.get
RUN mix compile

However, that renders caching completely useless, since any change to any file will invalidate the COPY step, and wll force all deps to be fetched again.

A common workaround is:

# copy only the stuff mix requires to work
COPY config/ mix.exs mix.lock ./
# fetch deps, and compile them
RUN mix deps.get
RUN mix deps.compile
# copy everything else
COPY . .
RUN mix compile

This way, when changing app code, we can still enjoy the docker cache that contains our compiled dependencies.

However, to do the same thing with an umbrella app requires us to get our hands dirty, since mix requires all the config files and mix.exs's to work:

COPY config/ mix.exs mix.lock ./
COPY apps/app1/mix.exs apps/app1/
COPY apps/app1/config/* apps/app1/config/
COPY apps/app2/mix.exs apps/app2/
COPY apps/app2/config/* apps/app2/config/
...


It would be nice if we could have a task that fetched dependencies based on our mix.lock alone. Something like `mix deps.get --deployment` or even a completely different task altogether.
I tried to hack the elixir repository to see if this was feasible, but it seems it would require deeper changes to how dependencies are fetched.

Michał Muskała

unread,
Jun 12, 2018, 6:50:17 AM6/12/18
to elixir-lang-core
I think the problem here is configuration. While mix.lock would probably be fine for fetching dependencies, it's definitely not enough for compiling them. Mix config through macros can affect how a dependency is compiled - the setting where to find config files is present in the mix.exs file, so it is definitely required for compilation. Additionally, it can also define some extra things like defines for the erlang compiler or env variables for make, etc. Because of that, I'm not sure it would be very useful to remove the requirements of mix.exs.

Michał.
--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/039b61c2-3dcd-48b0-8248-d473c34289e5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Allen Madsen

unread,
Jun 12, 2018, 8:22:14 AM6/12/18
to elixir-l...@googlegroups.com
The step after the one Miguel is describing would copy everything including the mix.exs file and then compile. So the configuration issue you mention shouldn't matter.

The sequence would be:

1. Copy mix.lock
2. Pull deps
3. Copy everything else
4. Compile

That order allows deps to be cached if the mix.lock hasn't changed.


To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/51cf60ed-9e32-442e-b585-e16fb0a08ca1%40Spark.

Norbert Melzer

unread,
Jun 12, 2018, 8:27:22 AM6/12/18
to elixir-l...@googlegroups.com
This will again cause recompilation of the dependencies, as soo as a file in the project has been touched.

But I do not see why one need to copy all config files in an umbrella.

If I had such a scenario, I'd centralise my config under the umbrella and remove the line that loads the configs from the embedded projects.

Or drop the umbrella overall, to make it easier to deploy separate applications to separate containers, making horizontal scaling even easier...

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAK-y3CvNrrbus_KBFzZpG_fhygmd5kj4Hbybb9AZT%3Dzi7u1vtg%40mail.gmail.com.

Allen Madsen

unread,
Jun 12, 2018, 10:03:03 AM6/12/18
to elixir-l...@googlegroups.com
I did completely miss that compilation was happening directly after fetching the deps in the original post though.

I was thought it was asking about being able to cache the fetch of the dependencies, not the compilation.




To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAK-y3CvNrrbus_KBFzZpG_fhygmd5kj4Hbybb9AZT%3Dzi7u1vtg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CA%2BbCVsuxDOActzb5ooPSJPad8ffY%2BbanatAY_N52okDD-tMMow%40mail.gmail.com.

Miguel Palhas

unread,
Jun 12, 2018, 10:17:40 AM6/12/18
to elixir-l...@googlegroups.com
I did forget that compilation required config files from the app.
My original thinking was about caching the dependencies yes, I added the steps for compilation as well in the example, since that's what I'm doing on one of my apps.

I guess with that in mind, adding a solution only for fetching deps feels more like a monkey patch, rather than a real feature?

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAK-y3CvNrrbus_KBFzZpG_fhygmd5kj4Hbybb9AZT%3Dzi7u1vtg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CA%2BbCVsuxDOActzb5ooPSJPad8ffY%2BbanatAY_N52okDD-tMMow%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAK-y3Ctp8AnDFzspqbW5%2BM%2BYH38t7Opu8pwWf2mDJfGEpqzhjA%40mail.gmail.com.

Paul Schoenfelder

unread,
Jun 12, 2018, 12:28:53 PM6/12/18
to elixir-l...@googlegroups.com
I think what you should be using here are Docker's multi-stage builds - they are designed to solve this exact problem. This is even more useful when you realize that for some web applications you need all the Node tooling and anything those depend on in the base image, as well as all the sources (and the notoriously large node_modules). Using multi-stage builds, your final image doesn't need any of that, just the Erlang/Elixir runtime.

Paul

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAEmrurpajX9mgMpu0Ny33fu9raafS%3DMPrqrELxBQ4gvQEBR7KA%40mail.gmail.com.

Miguel Palhas

unread,
Jun 12, 2018, 12:38:17 PM6/12/18
to elixir-l...@googlegroups.com
I am using a multi-stage build, actually. But I'm failing to see how that could solve my problem

What I was aiming for here was to avoid this:

COPY . .
RUN mix deps.get

because this prevents any kind of caching to exist in the deps.get stage, since a change to any single file on the codebase invalidates the cache on the previous step.
This would save a few seconds of every single build in our CI

This is not about reducing the final image size, which yes, multi-stage builds do help with. But maybe I'm missing something in your suggestion?

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAEmrurpajX9mgMpu0Ny33fu9raafS%3DMPrqrELxBQ4gvQEBR7KA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAK%3D%2B-Tt_cQWFWnXGXA_HZJQt8smJPP60iDaDzOr2q6d7a-%2Bjaw%40mail.gmail.com.

Paul Schoenfelder

unread,
Jun 12, 2018, 1:19:35 PM6/12/18
to elixir-l...@googlegroups.com
Ah sorry, yeah I missed the real problem you were hoping to solve, which is the cache. I thought using wildcard paths in the COPY might work to ease the burden, but I forgot that the destination is not very flexible, so it doesn't really work for this case.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAEmrurpajX9mgMpu0Ny33fu9raafS%3DMPrqrELxBQ4gvQEBR7KA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAK%3D%2B-Tt_cQWFWnXGXA_HZJQt8smJPP60iDaDzOr2q6d7a-%2Bjaw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAEmrurqXHM8JnNnUvH15qiheahZhjS2uBwmk91PuFbvpjYSk5A%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages