[erlang-questions] is sys.config mostly a convention?

[Xavier Noria]

Some explanations of what's a release kind of assume there is a sys.config file to override application environment variables.

I don't quite find source code in Erlang/OTP that specifically looks for that file. What I see, instead, is that the scripts generated by build tools pass -config sys.config to erl(1) if the file exists. Then, the application controller merges whatever goes in -config (though some comments in the source code do mention sys.config).

So, is sys.config a special-cased file in releases like other expected directories or files? Or is it actually just a convention?

I noticed "Designing for Scalability with Erlang/OTP" does not mention sys.config when explaining releases. It only appears later on together with vm.args when talking about rebar3.

[Loïc Hoguin]

On 02/13/2018 09:11 PM, Xavier Noria wrote:
> Some explanations of what's a release kind of assume there is a
> sys.config file to override application environment variables.
>
> I don't quite find source code in Erlang/OTP that specifically looks for
> that file. What I see, instead, is that the scripts generated by build
> tools pass -config sys.config to erl(1) if the file exists. Then, the
> application controller merges whatever goes in -config (though some
> comments in the source code do mention sys.config).
>
> So, is sys.config a special-cased file in releases like other expected
> directories or files? Or is it actually just a convention?

It does not sound like it's only a convention based on this part of the
documentation: http://erlang.org/doc/man/config.html

For example: "When installing a new release version, the new sys.config
is read and used to update the application configurations."

I do not know if that is truly representative of what the code is doing,
though.

Cheers,

--
Loïc Hoguin
https://ninenines.eu
_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

[Xavier Noria]

I have done a walkthrough, and the only thing that I see special-cased is that sys.config has hard-coded support for nesting.

One of the first things that the application master does when it starts is to check `init:get_argument(config)`, which simply lists all the files passed to `-config` files. The `init/2` function iterates that list and *If* the basename of one of items is "sys.config", then we enter that `if` branch. But the implementation does not assume its existence. In particular, it does not assume its existence regardless of whether the mode is interactive or embedded (which is said in the config docs).

I confirmed with a sample release, changed the value of `-config` to point to a dummy foo.config, the system boots normally, and `init:get_arguments(config)` indeed returns

    {:ok, [['/Users/fxn/tmp/sample/_build/prod/rel/sample/foo.config']]}

So my conclusion is that it is really a convention (except for the fact that the documented nesting does not work in arbitrary config files).

[Xavier Noria]

On Wed, Feb 14, 2018 at 10:24 AM, Xavier Noria <f...@hashref.com> wrote:

One of the first things that the application master

*application controller 

[Richard Carlsson]

Like an operating system, Erlang has several levels. The runtime system boot stuff assumes very little about standard libraries at all, and what you're going to do with any flags that don't affect the runtime system directly. Then the kernel and stdlib basics adds some conventions but still don't assume too much about overall system behaviour. Then, the application controller adds conventions about applications and -config (but lets you have as many config files as you like, with arbitrary names). Then, the release handler and reltool, if you use them, add further conventions, such as "you should only have one main config file". (For historical reasons, you may also find abstraction leaks between these layers.)

        /Richard

[Xavier Noria]

Even more, I see distillery supports an OS environment variable SYS_CONFIG_PATH to be able to point to an arbitrary config file (whose basename can be anything). I wonder if that may be handy for production deployments in which the machine that generates the release does not have access to sensible config data.

[Xavier Noria]

On Wed, Feb 14, 2018 at 11:13 AM, Richard Carlsson <carlsson...@gmail.com> wrote:

Like an operating system, Erlang has several levels. The runtime system boot stuff assumes very little about standard libraries at all, and what you're going to do with any flags that don't affect the runtime system directly. Then the kernel and stdlib basics adds some conventions but still don't assume too much about overall system behaviour. Then, the application controller adds conventions about applications and -config (but lets you have as many config files as you like, with arbitrary names). Then, the release handler and reltool, if you use them, add further conventions, such as "you should only have one main config file". (For historical reasons, you may also find abstraction leaks between these layers.)

Thanks Richard.

Yes, I understand this has to be like an onion, with the inner levels providing generic support and assuming less, and the outer levels enforcing more and exposing the actual contract. Also, I guess documentation and implementation have gone through a lot of evolution with all its practical implications.

In this particular case, though, take this:

"When starting Erlang in embedded mode, it is assumed that exactly one system configuration file is used, named sys.config."

I believe that wording is not accurate, because I see actually no code assuming that. Inner, or outer! Don't you think that something like

"By convention, releases are expected to have one system configuration file named sys.config stored in such and such."

?

[Peti Gömöri]

hi Xavier

the rebar2 start script template also had a section "Use releases/VSN/sys.config if it exists otherwise use etc/app.config"

https://github.com/rebar/rebar/blob/2429ddf7b46597222b0eaf6b79a438baafaaa96f/priv/templates/simplenode.runner#L82

Exactly as you say this is useful to package the configuration (etc dir) separately from the release (which does not contain a sys.config)

On Wed, Feb 14, 2018 at 11:13 AM, Xavier Noria <f...@hashref.com> wrote:

Even more, I see distillery supports an OS environment variable SYS_CONFIG_PATH to be able to point to an arbitrary config file (whose basename can be anything). I wonder if that may be handy for production deployments in which the machine that generates the release does not have access to sensible config data.

[Xavier Noria]

Thanks Peti!

That adds confirmation to the mental model I am building.

All squares, I think:

* OTP as such does not assume anything, despite what the config docs say.

* Release or not, embedded or interactive, the interface is just -config.

* Each one of the release tools is responsible for the -configs they generate.

* Build tools may support a default that can be overridden, up to them and for them to document.

* By convention, sys.config is a common default choice you'll normally find.

* Files called "sys.config" may include other config files.

[Tristan Sloughter]

It is not a convention. Only with a file named sys.config will the runtime read in additional configs you may include within it by name. With sys.config you can do:

[{my_app, [...]},

 "other.config"}].

This will inclue the content of other.config, but only if the file this is in is sys.config.

Note that I have a PR to also make sys.config.src an option, https://github.com/erlang/otp/pull/1560. This improves the ability to template a config, like is done with RELX_REPLACE_OS_VARS. Currently your template has to be a valid Erlang term, so `{port, ${PORT}}` doesn't work, only `{port, "${PORT}}.

--

  Tristan Sloughter

  "I am not a crackpot" - Abe Simpson

  t...@crashfast.com

[Xavier Noria]

On Sat, Feb 17, 2018 at 7:49 PM, Tristan Sloughter <t...@crashfast.com> wrote:

It is not a convention. Only with a file named sys.config will the runtime read in additional configs you may include within it by name. With sys.config you can do:

[{my_app, [...]},

 "other.config"}].

This has already been stated in previous messages, and it is implemented here.

If you look at the code, it says that *if* the basename is "sys.config", you *can* nest. That is not the same as saying that you need sys.config to load an application in a release. We have seen some counterexamples, you can edit the generated shell scripts to load ~/foo.config and it boots just fine, some tools have support for alternative files, and, definitive, if you read application_controller.erl you'll see the file is not assumed to be anywhere.

The release handler is the only piece of code to my knowledge that critically depends on its existence. If I read its code correctly, if you want the environment variables merged in an upgrade, sys.config *must* be located under Vsn. *That* is a requirement, but if you don't do release upgrades, anything can be passed to -config (which is what generated scripts do).

[Tristan Sloughter]

Right, you don't need it to load an application in a release. But that wasn't your question. I was just replying that it isn't a convention since specific features, as you note, rely on it.

--

  Tristan Sloughter

  "I am not a crackpot" - Abe Simpson

  t...@crashfast.com

[Éric Pailleau]

Hi,

Yes,  but using relative paths to another config file included, does not give same result depending if used in a release or start manually with -config. Path is from cwd (working directory)  which is not the same in both case. 

Very annoying. 

Regards 

[Éric Pailleau]

I mean that relative path in sys.config should be relative to sys.config itself, not depending to another unstable value...