Yesod 1.4 reminder

[Michael Snoyman]

Just a reminder: we're going to be releasing Yesod 1.4 Real Soon Now(tm). I'm talking very likely in the next week. If you have any patches you want to make it in, now's the time; it'll be very hard to convince me to do a breaking change for a while after this release.

I'm a bit torn on making the scaffolding changes I discussed in my blog post regarding environment variables, since there seems to be quite a bit of opposition to it. I still think it's the right change, but if the current setup is working as a good compromise solution, I'm not necessarily inclined to rock the boat. If you have an opinion either for or against the proposed environment variable changes, please write back to this thread.

[Pat Brisbin]

On Wed, Sep 10, 2014 at 09:04:42AM +0300, Michael Snoyman wrote:
> If you have an opinion either for or against the proposed environment variable
> changes, please write back to this thread.

I'll admit, I haven't been following the discussion so I'm not sure what the
reason is for the opposition to environment variables. When I first saw the note
about it as a potential change I was happy. Configuration via environment
variables is an important tenant of 12-factor[1] and I also happen to find it
quite pleasant in general.

In my open source apps, configuration is often split with things that can be
committed to source code and things that can't. Therefore I have to use[2]
environment variables anyway, so doing everything in that way would be a
simplification for me.

I was also going to mention a library I made to make env-based configuration
easier[3]. It's a port of the dotenv gem used in Rails. Since it's annoying to
ensure certain environment variables are set on your development or testing
boxes, it looks for a ./.env file (kept out of version control) and parses it
for variables to be set. Projects can ship a .env.sample with default or dummy
values to help with bootstrapping. It's quite nice.

Anyway, I thought I would make a vote in favor and share that library in case it
addresses any inconveniences raised by others.

1: http://12factor.net/config
2: https://github.com/thoughtbot/carnival/blob/master/Application.hs#L111
3: https://github.com/pbrisbin/load-env

Thanks,
Pat

--
patrick brisbin

[Hugo Gomes]

I am against the use of environment variables.

Environment variables are intended to provide a simple way to share configuration settings between multiple applications and processes.

However I would prefer not to use environment variables instead of a configuration file to configure the output of Yesod because:
  1. For complex configurations the amount of variables can increase, making the whole configuration difficult to read.
  2. The environment is shared by all applications under the same user, making running multiple instances more difficult.
  3. The environment can be changed by many things like differences between SSH interactive and non-interactive sessions, making it likely to suffer from unintended side effects.
  4. The environment is defined at process start, making it hard to reload the configuration without restarting the process.
  5. It is not clear how to setup environment variables in a OS-agnostic way (in *nix it changes even between shells).
  6. They don't allow for proper permission handling and cannot provide an easy way for different roles of colleagues to configure/restart the process without actually owning it.

Using configuration files seems much easier, more flexible and less prone to problems than environment variables. They can also be considered OS-agnostic as long as there is an abstraction of the file system path. Personally I think we should be moving towards having a good configuration manager (with files, or probably a mix of files/environment varialbes if necessity demands it) instead of a single configuration design.

Also bad handling of a repository must not constitute an argument in favor of environment variables because creating and handling a repository has to be something independent of an open source project design and human error can only be contemplated to a certain extent. Things like the most common use case of the scaffolded site MUST have a higher precedence in order to make this kind of decisions (and I could argue that such use case is NOT something that must be "suitable for deployment on modern cloud platforms, obviating the need for servers and systems administration" as the 12factor page says).

After reading a bit of the 12factor page I have some difficulty in understanding why is it being so regarded as most of the argumentation there seems one-sided and rather weak by being too overly specific in order to have the generic character that it attempts to achieve. But this is not the focus of this subject and could probably be discussed with the person that wrote that page instead.

All of the above are just my opinions on this subject.

[Greg Weber]

The config file approach we have now is still going to operate the same way it does now. I think we would just like to make it easier to configure via a database or ENV variables, and one possible approach to that is to always inject configuration through env variables.

--
You received this message because you are subscribed to the Google Groups "Yesod Web Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to yesodweb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Hugo Gomes]

That is an argument in favour of building a yesod configuration manager.

[Vagif Verdi]

As i suggested earlier on reddit, we could build a API for configuration plugin and provide out of the box 2 standard plugins: yaml file (default), env variables (alternative), with a possibility to add more in the future (db plugin, windows registry plugin etc).

And scaffolding app can as a questions how to handle configuration.

[Nolan Darilek]

Why not have a syntax for embedding environment variables in the config?
$environment.VAR, for instance? I'm certainly not proposing *that*, just
something like it.

I can see benefits to both approaches. All of my apps are deployed in
Docker containers, which negates most of the arguments against
environment variables, but they're certainly still valid in
non-containerized instances. Environment variables would make
Docker-based configuration easier, but it certainly isn't universally
appropriate.

[David Thomas]

"All of my apps are deployed in Docker containers, which negates most
of the arguments against environment variables, but they're certainly
still valid in non-containerized instances."

Is there something I'm missing here? What's so hard about environment
variables outside containers? Hugo Gomez earlier said "The

environment is shared by all applications under the same user, making

running multiple instances more difficult." but that seems false -
every application carries its environment and new processes inherit
from their parent...

I suppose if you're trying to fork things with different environments
from different threads in a single process things could be awkward...
is there something else I'm missing?

[N/A Jamesb]

I've been quite happy with configuration files, they are easy to read, saved in CVS, portable, etc.

I don't mind to see environment variables support in extra. I think it can be a good idea to just allow specifying these environment variables into the config file directly (like shell expansion). I don't think I'll use of though.

The problem with environment is being volatile, how would you set them consistently for your project? I hope it's not a customization to do. Moreover what about types? It's only string whereas configuration such as yaml supports into, etc.

[Maxim Kulkin]

I want to say a word in defense of environment variables. I'm using Yesod to develop application that is deployed on PaaS (Heroku). To deploy there (if anyone is not familiar with Heroku) you need to push your code/binaries to their Git repository and they will run instances based on that. It is very inconvenient to toss back and forth configuration files (as every change to config file requires a separate Git commit). However, they provide a nice CLI/API to manage configuration settings that end up as an environment variables visible to your processes.

We plan moving away from Heroku to our own infrastructure and packaging applications into Docker containers. With Docker, configuration based on environment variables is also more convenient than modifying containers for every config change.

[Michael Snoyman]

It seems clear to me that neither environment variables nor config files will make everyone happy at the same time. So let's see if we can come up with some middle ground, inspired by Vagif's comment.

We currently have two config files, encompassing three sets of configuration:

* config/settings.yaml has some built-in settings (listening port, application root) and user-configured settings (extra).

* config/postgres.yaml (or equivalent) has database configuration values.

Both of these files allow for switching between various versions of the settings within the file based on the command line argument indicating the environment type, e.g. development, production, etc.

How's this for a strawman proposal:

1. We simplify config files so that they only have one version of the settings.

2. We collapse the two config files into a single config file.

3. We make it easy to programmatically switch which config file is used, to emulate the current multi-environment setup.

4. We additionally make an easy setting to ignore all config files and grab data from the environment, with some extra fanciness to switch that based on some development environment variable to simplify when working with `yesod devel`.

5. We add documentation to the config file explaining how to use environment variables instead of the config file, and mark up the scaffolded code likewise.

I *think* this is just an elaboration of what Vagif said.

[Greg Weber]

Are 1-3 a proposal to not have separate environments in a config file? The shared default aspect of YAML configuration is very convenient and having a separate default file would make things more cumbersome. I think the idea of a configruation manager is that we don't necessarily have to change anything about how the config files currently work (unless we want to for the sake of the config files) and the configuration manager can be set to use either config files or env variables.

[Michael Snoyman]

OK, so the config manager would be capable of saying "OK, take this config file, and then grab the "development" key in there, and treat it as top-level," right?

[Greg Weber]

I was thinking that would be the job of one of the configuration providers for the configuration manager. There could also be a configuration provider that did not look for different environments in the YAML file.

[Vagif Verdi]

Both yaml files and env variables are just a form of storage.Just like persistent works exactly the same way be it postgress of mysql, we should have a configuration interface that allows to plugin any specific storage, be it a yaml file or env variables, or in the future windows registry or LDAP service.
The API should allow the same operations across all kinds of configuration storages. So if you want to read different set of properties based on the executable parameter (Dev, Production, Staging etc) then all configuration storages should support that in the same manner.

It would be ideal if this change will be transparent for current yesod deployments or at least would result in minimal changes to accommodate new configuration manager while letting current applications to continue to use their yaml config files as is.

[Alexandr Kurilin]

Something people do in the Clojure universe with libraries like environ is have a layered approach: there are defaults for every configurable parameter, which are overridden by the defaults for a specific environment, which are overridden by whatever is defined in env vars (you'd use envdir against a folder of env var files when running the app). Thus you can have universal defaults, environment defaults, and still trump everything, if you need to, when you're deploying through configuration management and you have very specific specs.

[Michael Snoyman]

I think what this system really needs is someone to volunteer to guide the process, since there are clearly many people out there with far stronger opinions on how this should look than I have. Does someone want to step up and design the next-generation Yesod configuration management system?

[Greg Weber]

Rather than a configuration typeclass we may be able to focus on using combinators.

If we look at the csParseExtra interface to loadConfig (it lets a user load an arbitrary Extra configuration) it looks like this:

parseExtra :: env -> Object -> Parser Extra

Rather than pre-loading YAML as JSON it can let the function do that:

loadExtra :: env -> IO Extra

That would let you change it to use YAML, env vars, or whatever. The only problem now is the concept of overriding: using a config file but then overriding with env vars. Then we really need 2 forms of combinators. The above, to load in everything, and an additional form just for modification:

modifyExtra:: env -> Extra -> IO Extra

It would be nice to separate out IO from this as parseExtra did, but parseExtra decided to use Object as an intermediate representation, and I am not sure if that is appropriate or not for every configuration source.

[geoff]

I know I am late to the discussion, but my 2c:

I think adding environment variables while leaving everything else the same is great.  Users who don't want to use them have full backwards compatability, while users who want them can use them.

Ideally, the reference to the environment would go into the configuration file itself, something like like [1]

Default : &defaults

  host: "*4"

  post: <%= ENV['YESOD_APP_PORT'] %>

1: https://stackoverflow.com/questions/5866015/rails-3-how-use-an-env-config-vars-in-a-settings-yml-file