Django's build: Try Earthly?

[Vlad A. Ionescu]

Hi Django devs,

Nick Pope directed me to this list. Wondering if anyone is interested in trying https://github.com/earthly/earthly for Django's build.

Earthly could help with reproducing CI failures locally (via containers), testing against multiple versions of Python in parallel, or for migrating between CI vendors.

It works well on top of Jenkins and GH Actions.

Happy to put together a demo PR if this is of interest. Let me know!

Cheers,
Vlad.

[Tom Forbes]

Hey!

I've recently been trying to improve our local/CI experience so this is quite a nice and timely message!

I really like the look of Earthly and it has some uses at $WORK that I'm excited about, but for Django the most annoying thing is the large build matrix rather than a complex dockerfile. I really dislike docker-compose for this, but I can't see anything in Earthly that would help manage or reduce this yaml-spaghetti?

Tom

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAO_OUmM%2BMFU1nPMzFaEsGcD4rh6tFh-wQ3T_hK4GH5-4%2BCxgxQ%40mail.gmail.com.

[Vlad A. Ionescu]

Hi Tom,

I believe that a lot of what django-docker-box achieves is what Earthly was created for too. Now that you already have django-docker-box, it’s possible that you might not need Earthly. Or at least not for the purpose of reproducing CI integration testing locally, specifically.

In any case, here’s how Earthly might help. Emphasis on might, as I have limited understanding of the complete Django dev ecosystem - I’m pretty much shooting in the dark here :-)

• Beyond CI integration testing, if there are other CI scripts, Earthly can help containerize those too in order to achieve reproducibility. It can help debug builds faster on your own computer, rather than wait for CI to kick in again and again.
• If django-docker-box is placed in Earthly, then Earthly could run these tests isolated and in parallel. e.g. Run Python 3.7 tests in a docker compose stack distinct and isolated from a Python 3.8 tests stack. Similarly for different DB versions. Every matrix tuple combination can be its own isolated execution that runs in Earthly in parallel.
• In terms of yaml spaghetti: it may be possible to decouple some of the integration tests into distinct docker-compose yaml files which are executed separately through Earthly. It may or may not be beneficial to trade-off yaml spaghetti with some Earthfile scripting to help tame the complexity. On one end of the spectrum you could get rid of all the docker-compose scripting and use docker run inside Earthly. On the other end of the spectrum, you can just run compose stacks in Earthly itself as we support that pretty well. Elixir’s Phoenix framework, for example uses docker run‘s for DB waiting, docker compose for dependencies and mix (Elixir’s build tool) for running the tests. This is what this looks like in practice..
• In general, when migrating from one CI vendor to another, migrating to Earthly first helps abstract away the CI. Earthly runs the same on either Jenkins or GitHub Actions, or your own laptop. This helps with having a common CI language that works the same regardless of the vendor.
• Compute intensive steps (if there are any) can be cached and not repeated if the inputs have not changed. In Earthly this can work with a remote cache too.
• Earthly’s multiplatform features could run tests on other architectures too (e.g. arm64), in parallel (uses QEMU underneath).

If any of this feels interesting, I’d be happy to dig deeper and put together a demo PR to POC a particular benefit.

Cheers,
Vlad.

You received this message because you are subscribed to a topic in the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-developers/BXxm2h-9ECg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFNZOJM1SEpJiJdsq0AFmOefZFRBPmaDXZseE3e4J7QALw%2BsKw%40mail.gmail.com.