Proposed overhaul to the scaffolding

[Michael Snoyman]

I just spent quite a while working on this, and it's late at night, so apologies if this email is a bit scatter-brained.

As some of you may remember, I proposed overhauling the Yesod scaffolding- and in particular the configuration system- as part of the Yesod 1.4 release. After some debate about environment variables vs config files, I dropped the issue. However, since this issue has bit me multiple times since then, I decided to take a real crack at it. These changes are now on the settings-cleanup branch[1].

There are *lots* of changes on that branch (26 commits currently). So instead of getting into details immediately, let me describe the goals at a high level:

* Make a more sensible divide between scaffolding and library functions. This should make it easier to extend the scaffolding, while removing some unnecessary boilerplate.

* Have a single setting data type for the application which contains all possible settings.

* Let that setting data type be read from: config files at compile time, config files at run time, and environment variables.

* Make all three of those features configurable in the scaffolding.

Now to jump into the approach. That single setting datatype is visible at:

https://github.com/yesodweb/yesod-scaffold/blob/f12e7471e07517dceda72c2019dd161b90f9fdd0/Settings.hs#L29

One thing that I did away with here is the "development" variable we used to have. Instead, there are individual configuration values for various things that were previously controlled by development. This finally gets us away from the bad situation of needlessly conflating the word "development" with things like "what kind of logging should I use?" All of those values take a default based on whether you're using `yesod devel` or compiling to an executable[2], but that default can be overridden.

The next important thing is to see the YAML file that feeds that settings type:

https://github.com/yesodweb/yesod-scaffold/blob/f12e7471e07517dceda72c2019dd161b90f9fdd0/config/settings.yml

The special thing here is the structure `{env: FOO, value: BAR}`. This is the magic that lets us "have our cake and eat it too." It means "if the environment variable FOO is set, use its value, otherwise use BAR." For those of you looking to ignore environment variables and just use config files, you just need to change a True to a False[3]. For those looking to require all configuration be provided via the environment, you just need to turn off the default compiled-in configuration[4].

While the env: ... stuff is a little tedious, I think it's a big improvement. Many people have been confused previously by implicit overriding of things like APPROOT and PORT. This makes the whole process very clear: anything in this config file can be overridden by an environment variable in a predictable way.

You can also have multiple configuration files, with each one shadowing values from the previous one. For example, test-settings.yml[5] modifies the database name used in the test suite. This allows you to keep the majority of your settings in a primary config file, and just override the necessary ones for testing purposes (or, equivalently, for your staging environment).

I could go on about other tweaks made, but these are the important ones to discuss now. I've purposely avoided whitespace cleanups so far to make the merge a little less painful, so some parts look ugly; that can be fixed post-merge. We can also discuss moving from the ad-hoc Import to ClassyPrelude.Yesod, but that's a completely orthogonal discussion that I'd like to have *after* we discuss the settings system.

OK, brain dump complete :)

[1] https://github.com/yesodweb/yesod-scaffold/tree/settings-cleanup

[2] https://github.com/yesodweb/yesod-scaffold/blob/f12e7471e07517dceda72c2019dd161b90f9fdd0/Settings.hs#L65

[3] https://github.com/yesodweb/yesod-scaffold/blob/f12e7471e07517dceda72c2019dd161b90f9fdd0/Application.hs#L129

[4] https://github.com/yesodweb/yesod-scaffold/blob/f12e7471e07517dceda72c2019dd161b90f9fdd0/Application.hs#L126

[5] https://github.com/yesodweb/yesod-scaffold/blob/f12e7471e07517dceda72c2019dd161b90f9fdd0/config/test-settings.yml

[Michael Snoyman]

Oh, forgot to mention one thing. If you want to try this out, you'll need to install the `yesod` package from Github, since it includes a new module (Yesod.Default.Config2) which is used to implement a lot of the stuff in the scaffolding.

[Greg Weber]

For a less verbose syntax, take a look here: https://github.com/docker/docker-registry

common: &common
    standalone: true
    loglevel: info
    search_backend: "_env:SEARCH_BACKEND:"
    sqlalchemy_index_database:
        "_env:SQLALCHEMY_INDEX_DATABASE:sqlite:////tmp/docker-registry.db"

--
You received this message because you are subscribed to the Google Groups "Yesod Web Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to yesodweb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[geoff]

Michael

I really like this.  My only comment is that forcing an 'env' child when you may not want/use it doesn't make me that happy (perhaps I am just reading the code wrong.. in which case this is all for naught!).  

As a user, my preference would be for a less-strict settings file (perhaps still verbose in the scaffolding); and generate run-time errors if a value is not supplied when it is supposed to be.

Put another way, it seems *more* confusing for a user to have to go into Application.hs rather than just a config file to use/force environment variables.. whereas allowing config files that looked like..

host:
  env: MYAPP_HOST
port:
  value: 80
approot:
  value: localhost
  env: MYAPP_APPROOT

..would allow a someone other than the (or even, a) developer to intuitively know how to deploy things, and/or change how they are deployed.. without having to look at or change haskell code.

It might remain unclear to someone just looking at the settings file if value trumped env or viceversa; but generally if people can just remove one or the other to get the results they want it wouldn't really matter.

--

[Michael Snoyman]

I was worried about whether I could get it to work correctly with typed data, but it seems to be chugging along just fine. That's a *much* nicer syntax, and even better that it's following prior art. Thank you!

[Michael Snoyman]

On Thu Nov 20 2014 at 12:38:07 AM geoff <geoff...@gmail.com> wrote:

Michael

I really like this.  My only comment is that forcing an 'env' child when you may not want/use it doesn't make me that happy (perhaps I am just reading the code wrong.. in which case this is all for naught!).  

You mean: it should also be possible to have a setting in the YAML file without a corresponding environment variable? Yes, that's supported, both with the original syntax I had, and with Greg's improved syntax. For example, look at the "copyright" setting.

 

As a user, my preference would be for a less-strict settings file (perhaps still verbose in the scaffolding); and generate run-time errors if a value is not supplied when it is supposed to be.

Put another way, it seems *more* confusing for a user to have to go into Application.hs rather than just a config file to use/force environment variables.. whereas allowing config files that looked like..

host:
  env: MYAPP_HOST
port:
  value: 80
approot:
  value: localhost
  env: MYAPP_APPROOT

..would allow a someone other than the (or even, a) developer to intuitively know how to deploy things, and/or change how they are deployed.. without having to look at or change haskell code.

It might remain unclear to someone just looking at the settings file if value trumped env or viceversa; but generally if people can just remove one or the other to get the results they want it wouldn't really matter.

So I'm not certain, but I *think* this approach meets this design goal, but I'm not certain. My comment about Application.hs modifications is to give a developer full flexibility to completely ignore environment variables or config files. But the default would be to let you specify extra config files on the command line, and set environment variables to override things in the config file.

[geoff]

On Wed, Nov 19, 2014 at 5:57 PM, Michael Snoyman <mic...@snoyman.com> wrote:

You mean: it should also be possible to have a setting in the YAML file without a corresponding environment variable? Yes, that's supported, both with the original syntax I had, and with Greg's improved syntax. For example, look at the "copyright" setting.

Okay; I think I see what I misunderstood (I thought the values has to be of type {foo: {value: x, env:y}} even when ignoring env.. 

In any case, with the embedded _env with default it is basically perfect.  If the scaffolding will use the env by default; probably the env varriables should be in a yesod/app name space $YESOD_HOST $YESOD_PORT, etc.

This will make docker deployment even more straight forward

[Alexandr Kurilin]

Looks like a welcome change, I actually used to do something something very similar in a different language and it worked pretty well there.

Also exciting to see ClassyPrelude being integrated with Yesod, just learned about it today.

On a related note, I learned today that Foundation-less Import might be also a good idea for when you want Import, but none of Foundation stuff that comes with it. Is that something you folks might consider as well for scaffold refactoring?

--

[Michael Snoyman]

Geoff: PORT is already supported by Keter, FP Haskell Center, yesod devel, and Heroku, and APPROOT is supported by the former two. Besides breaking that compatibility, adding a YESOD_ prefix would mean tooling would have to specifically target Yesod, as opposed to having general-purpose deployment techniques. So I'm -1 on making the YESOD_ prefix the default, but the nice thing in this change is that it's now trivial to *change* the default.

Alexandr: What's the use case you'd want a Foundation-less Import for? It might be the same as just importing ClassyPrelude.Yesod. My concern about doing something like that would be trying to meet everyone's needs. Someone else might want all of Import, but without Foldable generalizations, for instance. It's unlikely that we'll be able to find a design point that solves everyone's desires at once. I think the design goal should be intelligent defaults and easy customization, while keeping things as small as possible.

[Greg Weber]

speaking of ClassyPrelude, are we ready to put it in the scaffolding at least as something to be uncommented?

[Michael Snoyman]

Going along with what I said to Alexandr: I'd rather the scaffolding just make a decision instead of trying to provide choices. Just providing the different database backends is quite a bit of variation, and trying to cross that with other arbitrary decision points could be very painful. So I'd like to say: either we use ClassyPrelude, or we don't. And whichever way we choose, let's document how to make the switch to the other on the Wiki, with a link in the scaffolding.

Now, answering the real question: I think ClassyPrelude is ready to be promoted to the scaffolding. I use it in all of my projects now, and I haven't run into issues. I also haven't heard complaints about it confusing people, especially since the move to mono-traversable. So that's a +1 from me. Any other views on it?

[Michael Snoyman]

One other thing: I'd like to set a deadline of Monday for merging in the configuration systems changes. I'm choosing a relatively short date since I haven't heard any objection to the changes yet. If there *is* objection, please bring it up sooner rather than later.

On Thu Nov 20 2014 at 7:22:48 PM Michael Snoyman <mic...@snoyman.com> wrote:

Going along with what I said to Alexandr: I'd rather the scaffolding just make a decision instead of trying to provide choices. Just providing the different database backends is quite a bit of variation, and trying to cross that with other arbitrary decision points could be very painful. So I'd like to say: either we use ClassyPrelude, or we don't. And whichever way we choose, let's document how to make the switch to the other on the Wiki, with a link in the scaffolding.

Now, answering the real question: I think ClassyPrelude is ready to be promoted to the scaffolding. I use it in all of my projects now, and I haven't run into issues. I also haven't heard complaints about it confusing people, especially since the move to mono-traversable. So that's a +1 from me. Any other views on it?

On Thu Nov 20 2014 at 6:04:18 PM Greg Weber <gr...@gregweber.info> wrote:

speaking of ClassyPrelude, are we ready to put it in the scaffolding at least as something to be uncommented?

On Wed, Nov 19, 2014 at 10:51 PM, Michael Snoyman <mic...@snoyman.com> wrote:

Geoff: PORT is already supported by Keter, FP Haskell Center, yesod devel, and Heroku, and APPROOT is supported by the former two. Besides breaking that compatibility, adding a YESOD_ prefix would mean tooling would have to specifically target Yesod, as opposed to having general-purpose deployment techniques. So I'm -1 on making the YESOD_ prefix the default, but the nice thing in this change is that it's now trivial to *change* the default.

Alexandr: What's the use case you'd want a Foundation-less Import for? It might be the same as just importing ClassyPrelude.Yesod. My concern about doing something like that would be trying to meet everyone's needs. Someone else might want all of Import, but without Foldable generalizations, for instance. It's unlikely that we'll be able to find a design point that solves everyone's desires at once. I think the design goal should be intelligent defaults and easy customization, while keeping things as small as possible.

On Thu Nov 20 2014 at 8:41:34 AM Alexandr Kurilin <al...@kurilin.net> wrote:

Looks like a welcome change, I actually used to do something something very similar in a different language and it worked pretty well there.

Also exciting to see ClassyPrelude being integrated with Yesod, just learned about it today.

On a related note, I learned today that Foundation-less Import might be also a good idea for when you want Import, but none of Foundation stuff that comes with it. Is that something you folks might consider as well for scaffold refactoring?

On Wed, Nov 19, 2014 at 10:23 PM, geoff <geoff...@gmail.com> wrote:

On Wed, Nov 19, 2014 at 5:57 PM, Michael Snoyman <mic...@snoyman.com> wrote:

You mean: it should also be possible to have a setting in the YAML file without a corresponding environment variable? Yes, that's supported, both with the original syntax I had, and with Greg's improved syntax. For example, look at the "copyright" setting.

Okay; I think I see what I misunderstood (I thought the values has to be of type {foo: {value: x, env:y}} even when ignoring env.. 

In any case, with the embedded _env with default it is basically perfect.  If the scaffolding will use the env by default; probably the env varriables should be in a yesod/app name space $YESOD_HOST $YESOD_PORT, etc.

This will make docker deployment even more straight forward

--
You received this message because you are subscribed to the Google Groups "Yesod Web Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to yesodweb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Yesod Web Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to yesodweb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Yesod Web Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to yesodweb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Yesod Web Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to yesodweb+unsubscribe@googlegroups.com.

[Christopher Reichert]

On Thu, Nov 20 2014, Michael Snoyman <mic...@snoyman.com> wrote:
> Going along with what I said to Alexandr: I'd rather the scaffolding just
> make a decision instead of trying to provide choices. Just providing the
> different database backends is quite a bit of variation, and trying to
> cross that with other arbitrary decision points could be very painful. So
> I'd like to say: either we use ClassyPrelude, or we don't. And whichever
> way we choose, let's document how to make the switch to the other on the
> Wiki, with a link in the scaffolding.
>
> Now, answering the real question: I think ClassyPrelude is ready to be
> promoted to the scaffolding. I use it in all of my projects now, and I
> haven't run into issues. I also haven't heard complaints about it confusing
> people, especially since the move to mono-traversable. So that's a +1 from
> me. Any other views on it?
>

I don't think usage of the library itself confuses people. Rather, I
think it's the feeling of having "another thing to learn" that could
discourage a person away from Yesod specifically as there are already a
ton of new concepts learn.

Good documentation could help mitigate this and I'd be willing to pitch
in as I test the new scaffold.

Would the changes needed to use classy-prelude even have a large impact
on the current scaffold? Or would it mostly be swapping out the imports?
Sorry, I have not used classy-prelude much yet but looking forward to
it!

I'm in favor of the change. +1.

Regards,

--
Christopher Reichert
irc: creichert
gpg: C81D 18C8 862A 3618 1376 FFA5 6BFC A992 9955 929B

[Michael Snoyman]

It's just an issue of imports. All of the handler modules and Application.hs just import the Import module, so we'd need to modify that to use ClassyPrelude.Yesod (which, frankly, will greatly *simplify* that modules currently hairy import list). Then we can similarly modify the other modules to replace `import Prelude` with `import ClassyPrelude.Yesod`, and likely get rid of a bunch of extra imports.

I'm hopeful that this change will result in almost no additional things for someone to learn. For the most part, identifiers from ClassyPrelude.Yesod are simply generalized versions of what Prelude or Yesod expose. For example, length in Prelude works on lists, and ClassyPrelude it works on any MonoFoldable (which includes lists). There are a few cases where there are new function names to learn (omap, insertMap, mapFromList), but those are all optional, and simply doing the "standard" thing (e.g., `import qualified Data.Map as Map` and `Map.fromList`) still works just fine.

Michael 

[Greg Weber]

I don't think the classy-prelude changes the scaffolding much. The main problem will be when someone uses a formerly partial function such as minimumBy and then they get an error about MinLen which they don't understand. This is a good thing but requires learning something new.

—
Sent from Mailbox

On Thu, Nov 20, 2014 at 9:41 AM, Christopher Reichert <creic...@gmail.com> wrote:

<signature.asc>

[Michael Snoyman]

Oh, good point, I completely forgot about those cases, I'm glad you brought them up.

--
You received this message because you are subscribed to the Google Groups "Yesod Web Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to yesodweb+u...@googlegroups.com.

[Michael Snoyman]

Given that there has been no objection to the changes we've been discussing, I've merged the changes into all of the individual scaffolding branches. I'll hold off on releasing these to Hackage for a few days so that people can test out the individual scaffoldings first.

To unsubscribe from this group and stop receiving emails from it, send an email to yesodweb+unsubscribe@googlegroups.com.

[Michael Snoyman]

OK, these changes are now on Hackage as yesod-bin-1.4.1. I'll publish a blog post on this in the next few days.

[Maximilian Tagher]

It'd be nice if the settings.yml file explained the _env:ENV_VAR:value convention. I came up with this, but it might be too wordy:

# Values formatted like "_env:ENV_VAR_NAME:default_value" can be overridden by the specified environment variable

# e.g. _env:PORT:3000 defaults to 3000 but can be overridden by the PORT variable

# Using environment variables gives you flexibility across multiple deployments; see http://12factor.net/config

# (If you don't want this behavior, just remove the _env:ENV_VAR_NAME: part)

To unsubscribe from this group and stop receiving emails from it, send an email to yesodweb+u...@googlegroups.com.

[Michael Snoyman]

What about making a wiki page with an explanation and simply linking from the settings file?

[Maximilian Tagher]

Yeah, that could work. I think if you're linking to something, I think you'd need at least a sentence to explain what you're linking to, like this:

# Values formatted like "_env:ENV_VAR_NAME:default_value" can be overridden by the specified environment variable

# See <wiki/blog post link>

Though honestly, just that one sentence might be enough, unless the wiki page wanted to get into the details of why environment variables might be used. I don't know if that's valuable information or overkill. 

[Michael Snoyman]

I think a wiki page explaining different ways of doing configuration would be a valuable addition. It can start small and be extended as people want to contribute new ideas. I think your two line addition would be perfect for the scaffolding.

[Maximilian Tagher]

Cool, I gave that a shot with this PR, which links to this wiki page.

[Michael Snoyman]

Awesome, merged.