Feature Proposal: passing variables to dockerfile (for use in custom configs).

[cyphar]

Currently, it seems as though the only way to have dockerfiles which have dynamic content is to run some seed file through a preprocessor and then send that to docker build. This looks something like this:

$ cpp -DVAR="value" ./dockerfile.in | docker build -

Personally, I find this design rather clunky and misguided. Does anyone have any problems with `docker build` to be modified to take another argument which will be used to populate variables in the dockerfile before processing the file?

My idea of the design is to follow the already-established-bashism of dockerfiles and make variable substitution work with $VARNAME.

Variables could be defined through:

1. Command-line parameters. (-D "VAR='value'")

2. Set inside the docker file. (SET VAR="value")

In case 2, each SET directive would only affect subsequent dockerfile directives. Of course, SET directives wouldn't need to be executed in an intermediate container, as they would not affect the docker image in any way.

Here is an example of how the directive would work:

$ cat dockerfile

# This is an example dockerfile which demonstrates variables

FROM ubuntu:14.04

RUN echo $VAR

$ docker build -D "VAR=hello world" .

# --snip--

Step 1 : RUN echo $VAR

 ---> Running in ...

hello world

 ---> ...

Successfully built ...

$ docker build -D "VAR=foo bar baz" .

# --snip--

Step 1 : RUN echo $VAR

 ---> Running in ...

foo bar baz

 ---> ...

Successfully built ...

$

This could be implemented either by:

1) Adding the SET directive as a special case in server/buildfile.go, which would update the build environment. Each replacement would be handled in BuildStep, before running the relevant Cmd_____

2) Creating a preprocessing stage in server/buildfile.go, where the entire file is preprocessed and then run through the build.

P.S. The function ReplaceEnvMatches from https://github.com/dotcloud/docker/blob/master/server/buildfile.go#L223 doesn't seem to be documented anywhere. It looks like variable substitution for ENV directives. Does anyone know what it does?

--

Aleksa Sarai

http://www.cyphar.com

[Michael Neale]

I suspect a key aim is to make "docker build" work as consistently as possible when someone gets it - so the result is close to what the person who wrote the Dockerfile intended (but I might be conflating that with the importance of portability of containers). 

[cyphar]

My understanding is that it is the purpose of docker images to be consistent, not the build scripts. The whole point of Dockerfiles is to automate the docker-related commands required to add each layer of the image and the configs. The problem with Dockerfiles being static is that many projects (my own included) require dynamic configuration of Dockerfiles in scripts. Since there is a need for this functionality, it makes sense (at least to me) to create a standard and default format by which it can be done. Besides, macro substitution isn't a novel or esoteric feature, it is a legitimate feature which most build systems provide. 

[cyphar]

I implemented this feature in #5624 (https://github.com/dotcloud/docker/pull/5624). I changed the name to macros, since they act more like macros than variables (due to their recursive definition).

[Geoffrey Bachelet]

Can you describe a few real world use-cases for this feature?

[Benjamin Wootton]

One area where this would help is versioning. E.g. if I want to arbitarily build a version for a source controlled tag 1, 2, 3, 4 then this could be a variable passed into the docker file.

I agree that as a principle you are best to have repeatable Dockerfiles AND images and configure containers at instantiation time but I think I'm a +1 on this change, especially labelled as macros so it's more a text substitution than an environment concept.

--
You received this message because you are subscribed to the Google Groups "docker-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to docker-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Johannes Ziemke]

What's the advantage over using a preprocessor? Given that you store your Dockerfiles somewhere and need to copy/download/check them out, you need some steps before running `docker build` anyways, so I don't really see why you can't just run a preprocessor as a additional step?

[Aidan Hobson Sayers]

Regarding "it is the purpose of docker images to be consistent, not the build scripts" - I'm not sure this is the case. My impression is that the focus is on reproducible builds and images come as a side-effect of that.

Dockerfile features that mean someone else can't download and immediately run the dockerfile have seen opposition in the past. Volumes were added only after a number of people spoke up for them - they break the reproducibility 'promise'.

I vaguely recall a note (that I can't currently locate) that dockerfile syntax should be kept minimal, and if it's possible to do something outside of dockerfiles then it should be.

In this case I don't see why the variable syntax needs to be part of dockerfiles - if you need it, then maybe something on top/instead of dockerfiles is appropriate (your cpp approach, or ansible/puppet/chef/shutit(see docker-user) if you want to get more complicated).

I find it easiest to look at dockerfiles as a build product themselves, an exact description of an image docker will build for you (not quite true, but that's an aside).

A bit like html - it defines the structure of content on a client machine, can be generated with templates and can be manipulated on the client side. But if you have the html for a page, you can unambiguously say what content will initially be on a user's screen.

Maybe things have changed, maybe I'm misremembering.

I do wonder where it might stop though - if we get macros/variables, do we get conditionals? ...which comes back to the right tool for the job.

To state biases: I don't use dockerfiles at all - I use one of the more 'powerful' alternatives I mentioned above :)

Aidan

[Bryan Murphy]

On Mon, May 5, 2014 at 2:20 AM, Michael Neale <michae...@gmail.com> wrote:

This could be implemented either by:

1) Adding the SET directive as a special case in server/buildfile.go, which would update the build environment. Each replacement would be handled in BuildStep, before running the relevant Cmd_____

2) Creating a preprocessing stage in server/buildfile.go, where the entire file is preprocessed and then run through the build.

A third option (what we use) is to generate your docker file using a templating language like erb (ruby) or jinja2 (python) and then build with the generated dockerfile.   This is really simple, and for companies like us who already use jinja2+yaml it plugs right into our workflow.  A few makefile rules and a simple python script and you are good to go.

Bryan

[cyphar]

The reason for this feature would be to make a *standard* method of creating dynamic build scripts. While I do agree that you can use external tools (and I do use external tools to generate my Dockerfiles for projects), it would make more sense for it to be standardised across all Dockerfiles. Otherwise, the whole concept of docker being "one binary you can drop anywhere" is broken since now you will have several flavours of Dockerfiles, each with its own dependencies.

I would like to note that this addition has backward compatibility in mind, where non-existent macros are untouched. This means that Dockerfiles which don't use the SET feature don't need be changed (unless they use '$$' -- this might be a potential issue, since $ is also used by bash for variables). For even more backward compatibility, it would be possible to change the syntax for macros so that there is no leading '$', which would make the macros akin to cpp, with no special terminals. The only drawback for using cpp-like macros would be that it would reduce the readability of macros, since they would be harder to spot when looking at the Dockerfile.

This only adds one directive and one concept (the SET directive and the concept of macro substitution). This *one* feature will allow for essentially any external computations (such as for loops, if statements, etc.) to be added dynamically to Dockerfiles. This one concept could be used in all areas of dynamic generation of Dockerfiles, which could massively improve the functionality of Dockerfiles with only _one_ addition to the syntax and concepts.

[cyphar]

Also, on the comments regarding repeatable builds: the builds *are* repeatable, if you give it the right macro values. This is functionally similar to passing env values, if you don't pass the right ones, the build won't be repeatable. The only difference being that the macros are both passable to `docker build` and are evaluated before running the step. This change has no effect on consistent images, only the build is changed. This has less of an effect than ENV on the entire docker process (since ENV both affects the build and run process -- but you can't specify them via the command line or use them as macros).

The main reason for me to add this feature is to essentially allow for modification of RUN calls to config and build scripts for the source of files, such as setting database usernames/passwords or taking generated "flag" values to be baked into the builds. All of these things can be done via an external tool that then pipes into docker build, but I feel that requiring an external tool for such a fundamental feature makes *no* sense.

[Gary Trakhman]

+1

My Dockerfile and related scripts creates a 'docker' user during the process.  I want to parameterize this based on the host system, for UIDs to match (for mounts). Disappointed this doesn't exist, now I will hard-code it.

Introducing a separate template language is more annoying than hard-coding it for now.

[Luis Faustino]

+1 

This proposed featured would be very useful.

One only uses it if wanted, so Dockerfiles without vars still work the same way.

[David Lewanda]

+1 as well. I hope the patch submitted above gets accepted. I have a similar scenario where we have a base container that we base multiple applications off of. The derived containers need a FROM line, but the FROM line needs to have a parameter to match the tag in source control from which it can be based. At a minimum we'd like to have stable and development versions of the base container, and be able to specify the proper revision in the FROM line in the derived containers depending on whether the derived container is stable or development itself. I hope this gains traction!

-- 

http://www.connectify.me/

This email message is for the sole use of the intended recipient(s) and may contain Connectify confidential or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not an intended recipient, please contact the sender by reply email and destroy all copies of the original message.

[Sean Fitts]

+1

I'd like to have Dockerfiles for installed various versions of 3rd party software where the version to install is controlled by the build environment, not the Dockerfile.  My alternatives now are to use an external template system or have version specific Dockerfiles.  I understand the goal of minimalism, but I think the usefulness of the feature out-weigh that in this case.

My only concern is potential overlap/confusion with ENV (which is already a bit funky in that sometimes it is expanded by Docker itself and sometimes by the shell in the container).

[Simone Aiken]

+1 for being able to use variables in Dockerfiles.

We have a different base image for each release for various reasons and have to update the tag of every Dockerfile every branch to point to the proper base for that branch.  We are automating it - but it is silly that we have to do so.  This wouldn't be a problem if your FROM is always pointing to a 3rd party baseimage but when you are creating hierarchies of containers internally it becomes a bigger deal.  

Our problem could also be addressed with a command line flag in docker build to specify the precise tag to use in combination with the FROM directive.   That way the Dockerfile could specify the base name of the FROM image and the build script could pull the release tag from the build environment.  

- Simone

[Ian Miell]

As I've mentioned before it is for similar reasons that I built ShutIt:

http://ianmiell.github.io/shutit/

pip install shutit

shutit skeleton

Dockerfiles' design is deliberately - and with good reason - limited in this way. If you want flexibility and configurability like this you might want to use ShutIt (designed for Docker), or another more traditional orthogonal config management tool like chef/puppet/ansible.

[Madhav Puri]

Hi Simone,

I think your use-case is a common one. Not to say it can't be handled outside docker but having a docker native way might provide a better alternative in certain situations.

There is an ongoing discussion in a docker PR #9176 . Please feel free to chime in the discussion and +1 or suggest improvements as needed.

Best.

--

Madhav

[Dreamcat4]

I've explained on PR #9176 that it breaks the existing trust establishment we have in place. You need to (at minimum) add some single line in dockerfile to say you are parameterizing certain ENV var(s). And declare which ones they are.

@duglin also raised a valid point that we also need a decent syntax for declaring the default value of those vars (if not set externally). I suggested |= which is otherise used in 'C' as a bitwise OR operator. Come up with a better syntax for that, and suggest a better --flag for the existing ENV command operator. And then discussions can move forwards past the current point.

[Madhav Puri]

hmm, I think there were already concerns raised around having to modify every Dockerfile in order to use variables. And PR #9176 in it's current form doesn't need Dockerfiles to be modified.

Anyways, let's not have that discussion here. I just wanted to point out that there is an outstanding PR that addresses the use-case brought up.

[Dreamcat4]

No that's incorrect information. Also it's relevant to point out that such PR will never be merged into official Docker for same reasons I've already highlighted. In fact it's already been rejected (effectively) by consensus of official Docker members.

Sure you can go privately go off and build your own un-official versions of Docker with whichever of these un-merges PR features compiled in. As you wish. However that's not going to help the general situation in public (regular users) without bringing all the other drawbacks of making a fork of Docker. Which for most affected people isn't worth it for sake of only 1 PR in isolation.