Proposal for pluggable template file systems

[Chris Keele]

This was a project I created a hacky proof-of-concept for over a year ago that I've picked back up, incidentally not inspired by the existing hex package. 

I'd intended to make it available as an archive, but as I started looking at the implementation of Mix.Tasks.Archive.Install I realized integrating with Mix.Local.Installer would greatly simplify it and enhance mix itself.

I figured I should pass this around before I get too deep into my fork of mix, which is already halfway feature complete (not including rewriting mix new to use scaffolding).

Mix.Tasks.Scaffold

Goal:

- Improve mix's ability to generate file systems in file generation tasks like mix new, mix phoenix.gen.x, and others.

- Standardize file system generation procedures to allow portability of such code.

- Finally, transparently allow mix users override project-provided file system templates with their own.

A scaffold can be any file, folder, or tarball intended to act as as a template file or file system of template files.

There are three clients of this code. To hoist some already-overloaded terminology:

- producers: projects that want to generate files, but opt-in allow users complete control of what gets generated

- consumers: developers using the producer project as a dependency that might want to customize the files it generates

- providers: developers that want to distribute packaged alternative scaffolds to consumers

Producers

Instead of using Mix.Generator by hand to generate filesystems, producers can specify a :scaffold setting in their project config.

The scaffold configuration is either a path to a template directory/file/tarball, or a list of two-tuples of scaffold name and path to corresponding template directory/file/tarball.

Example:

def project do

  [app: :phoenix,

   scaffold: [

     {"config", "priv/scaffolds/config.exs"},

     {"new",    "priv/scaffolds/new"},
     {"ecto",   "priv/scaffolds/ecto.tar.gz"},

    ]

  ]

end

When they want to generate files, for example by unpacking the ecto scaffold into the foo/bar directory, they can use:

Mix.Project.config() 

 |> Keyword.fetch(:phoenix)

 |> Mix.Local.fetch_scaffold("ecto")

 |> Mix.Generator.create_files("foo/bar", force: true, eex: [assigns: [foo: bar]])

If the project wants to defer to user-defined scaffolds, it can replace fetch_scaffold with get_scaffold:

Mix.Project.config() 

 |> Keyword.fetch(:phoenix)

 |> Mix.Local.get_scaffold("ecto")

 |> Mix.Generator.create_files("foo/bar", force: true, eex: [assigns: [foo: bar]])

Functions:

- Mix.Local.fetch_scaffold(:otp_app, "namespace" \\ nil)

Looks for scaffolds for the :otp_app base on its project config, completely ignoring local user scaffold installs. Uses namespace for apps with multiple scaffolds.

Returns path to a file, directory, or tarball so is suitable for consumption by Mix.Generator.create_files.

- Mix.Local.get_scaffold(:otp_app, "namespace" \\ nil)

Looks for scaffolds for :otp_app first in local user installs, and falls back to default defined in :otp_app project config. Uses namespace for apps with multiple scaffolds.

 

Returns path to a file, directory, or tarball so is suitable for consumption by Mix.Generator.create_files.

Namespaces are allowed to be relative file paths for namespace heirarchies. 

- Mix.Generator.create_files("path/to/thing", "target_dir" \\ nil, opts \\ [])

Constructs a file, directory, or tarball into target_dir, using nice Generator.create_file logging and overwrite prompts.

 

Opts include :force to not prompt on overwrites, and :eex to enable EEx interpolation.

- Mix.Generator.create_file("path/to/file", "source", opts \\ [])

Same as current implementation, but now checks for a :eex opt during processing.

           If :eex is truthy, it will pass both the filepath and the source through EEx before writing to disk.

If :eex is a list, it will use that as a binding during interpolation.

 

Consumers

 People using producer projects can continue on just as before. However, if they want to provide their own scaffolding, they simply put it in ~/.mix/scaffolds/:otp_app/namespace.

The Mix.Local.get_scaffold call in the producer project will find their local version and honor it.

Scaffolds can be files, folders, or tarballs. This supports several usecases:

- Keeping a personal scaffold for mix new

- Sharing git-distributed phoenix scaffold preferences inside an org

- Installing tarball scaffolds from remote sources

 

Tasks:

 - mix scaffold.install

Installs file, folder, or tarball scaffolds from local path or remote uri into ~/.mix/scaffolds/*

Remote uris must point to a single file or tarball; alternatively they can be git uris and will be installed as a folder.

Tarballs are left alone during install: they're only extracted during the actual file generation step. 

- mix scaffold

Lists installed scaffolds

- mix scaffold.uninstall

Removes installed scaffold.

 

Providers

Creating a scaffold for distribution is as easy as hosting a file, tarball, or git repository.

However, special support for projects that want to offer scaffold for other otp_apps is available through mix scaffold.build.

 

Tasks:

 - mix scaffold.build

Providers that want to distribute scaffolds as tarballs can use this task.

 

This mechanism should be preferred because it generates tarballs with the expected paths for consumption by Mix.Local.get_scaffold. 

It builds the tarball from the exact same :scaffold project configuration as producers.

 

The target :otp_app is specified via :scaffold_app. If not provided, it just uses the project :app name, making all producers able to run this command on their default templates out of the box.

 

Alternatively, multiple otp_apps can be targeted by providing a Keyword.t scaffold configuration of otp apps and scaffold configurations.

Enhancements

The phoenix generators currently rely on 4 different manually specified template generation modes: 

- :eex interpolate file

This is the default behaviour of a scaffold with eex enabled, as phoenix likely will always do. As proposed there is no way to disable this per file.

- :text raw file

This is the default behaviour of a scaffold with eex disabled. In order to emulate old phoenix behaviour, these should be rewritten with eex inside of them using eex quotations: <%% "<%= raw eex%>" %>

- :keep empty folder

This behaviour is totally unsupported by scaffolds, which operates entirely on mkdirp!ing a filepath Dir.name before writing to it. This is probably fine: scaffolds should include a .gitkeep in their stead.

- :append to existing file

This is the only behaviour totally unsupported with no workaround, used exactly once in phoenix for css.

Prompt for append:

This lacking could be remedied by augmenting Mix.Utils.can_write? to offer a third option: append instead of just y/n overwrite. Alternatively Mix.Utils.should_append? could be added, putting the user through two prompts for append behaviour.

This implies that passing force: true would eliminate any opportunity to append.

Maybe we could support additional force strategies: force: :caution for never overwriting, force: :append for pure addition, and even force: :conflict to generate git merge conflict syntax around the entire file instead.

Sprock it:

Another approach would be to have mix scaffold.build and Mix.Generator.create_files deal in middleware that parses certain intent-revealing file extensions a la Rails Sprockets, breaking portability and inviting doom to this RFC.

Summary

Everything proposed is 100% backwards compatible and opt-in; mix will behave as it used to in all existing calls to Mix.Generate and Mix.Local functions. Ie. no tests need to be modified in my proof of concept; only added. Projects that choose to use scaffolds will need to depend on at least the version of elixir that first shipped mix with scaffold support, so this should be incorporated into an elixir minor bump.

Everything should just work when presented with symlinks. Enhancements to Mix.Local.Installer, like ability to read SFTP or other uri protocols, propagate to scaffolds.

Experimenting with local scaffold handling kind of gears me up for returning to my previous, more ambitious mix proposal.

What do you think about this? Sufficiently useful to bring in to core? Opinions on rewriting mix new and phoenix to use it? Any other projects that might benefit from this sort of scaffolding?

[Chris Keele]

^ 

Keyword.fetch(:phoenix) should read Keyword.fetch(:app)