Chassis for mix meta tasks

[Christopher Keele]

There is a certain category of mix task that I think we can offer better tooling and automatic discovery for. This type of task is rarely parameterized, normally namespaced as a subtask, and generally serves as a setup/teardown step for a library at the filesystem level. I think of these as 'project lifecycle hooks'. Some examples include:

- new: create source files locally

- get: retrieve files across the network

- compile: create necessary code artifacts within the build folder

- build: create artifacts not necessary for the execution of the project or library's other tasks

- clean: remove artifacts that this library owns from the above 3 steps

- install: place files in a location such that they auto-enable some given functionality

- uninstall: remove files from such locations

Many libraries offer these sorts of task under their namespaces. The way these setup/teardown tasks work, I often find myself wishing that I could run all of a certain type at once as a sort of common 'phase' when interacting with a project. For example:

- before starting a parallelized CI run I want to `compile` everything I can into a cached build folder

- after passing a test run I want to produce all artifacts like project documentation, escripts, hex packages, test and documentation coverage reports, and the like

- after thinking I've fixed a warning emitted somewhere during compile time I want to force `clean` everything out so I can see a comprehensive list of remaining warnings

The `compile` task is a good example of elixir already sort of supporting this today. It's a simple meta-task that runs all `compile.xxx` tasks based on the `:compilers` configuration inside the project mix.exs.

I think we should run with this idea of a meta-task to the other lifecycle tasks. Optionally, I think we should allow for automatic discovery of them (such that configuration within the mix.exs is not required).

The proposal is simple: define a set of task names that should exhibit this behaviour, specify exactly what they should be responsible for as a guide to task implementers, and create meta-tasks for them that will enumerate the subtasks within their namespaces.

Libraries that want to opt-in to this framework can simply start naming their tasks under the appropriate meta-task namespace (ie. `mix build.dialyzer`) instead of their own (`mix dialyzer.build`).

Users that want to add existing tasks to a meta-workflow from libraries that have not opted in can just specify an alias in their mix.exs to place it within the namespace (ie `aliases: [{"clean.dialyzer", "dialyzer.clean"}]`).

Of the examples I listed above, `get` and `new` do not make much sense to run in bulk, as in order to discover other potential `get` tasks you need a compile phase after `deps.get`; and `new` is normally parameterized and rarely idempotent. `install` and `uninstall` also are generally specific one-off commands.

However, bulk compile, build, and clean tasks correspond elegantly to CI cache, on-success, and cache-cleaning phases respectively. If a common Elixir build script was defined in terms of these batch meta-tasks, it would be invaluable for projects that add or remove tooling, since their setup documentation and CI builds would keep working without modification.

Such meta-task tooling could be a starting point for identifying and unifying other common project lifecycle tasks across the ecosystem. I can envision a `report` meta-task separate from `build` that exists to produce human-readable reference files (like docs and test/documentation coverage reports) separate from machine-readable artifacts like PLTs, escripts, and hex packages; or a `push` meta-task that publishes hex packages, updates 3rd party code coverage sites, or triggers webhooks.

[Eric Meadows-Jönsson]

Compile needs a meta task because the compilers have a common interface, they affect each other and need to be wrapped up in task. The compile task does more than just run all tasks in the compile.* namespace.

The rest of the functionality you mention seems like they can be solved by aliases which should be more flexible. What do we gain by wrapping them up in meta tasks instead of using aliases?

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/cb7c4267-ffac-4a5c-a03d-91bcdaa52809%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Eric Meadows-Jönsson

[José Valim]

I also want to add that by pushing in this direction we may create the opposite problem: it will become hard to find all tasks that belong to a given library/framework. For example, if we adopted this pattern, phoenix tasks could spread over multiple namespaces and then trying to understand what Phoenix provides and how Phoenix affects a project becomes trickier.

So I agree with Eric, I prefer to keep the tasks under the parent namespace and then build my own meta tasks with aliases.

José Valim

www.plataformatec.com.br

Founder and 

Director of R&D

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAM_eaphBLvQ61F-%3D2ZSWgMfhWshsRVYdaNuW9aUraFykHz8Rrw%40mail.gmail.com.

[Christopher Keele]

These are compelling points. I also never thought through the interconnectedness of the compile tasks; I'm now inclined to agree that aliases are the way to go here instead of an implicit framework.

A middle ground occurs to me: what if, similar to the grouping of modules in the sidebar of ex_docs, we allowed regexes to play a part in an alias specification? This is still relatively explicit per-project while also allowing for a measure of auto-discovery of similar tasks.

[José Valim]

> we allowed regexes to play a part in an alias specification?

This kind of lookups would be very expensive, because we would need to traverse all load paths in the file system to get all modules names, convert them to tasks and see which ones match.

Plus there is always the chance you will get a task by mistake, causing things to misbehave or triggering further slow downs.

--

[OvermindDL1]

On the compile namespace thing, if mix compilers implemented a behaviour (easy to check for in the beam file) and they exposed a module attribute of what compilers should absolutely run before or after this one, then a dependency tree could be built of compilers in addition to running ones concurrently that do not depend on each other.  That'd be much nicer than asking the user to add our compilers to their mix files in every project.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/cb7c4267-ffac-4a5c-a03d-91bcdaa52809%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Eric Meadows-Jönsson