Supporting multiple environments like Rails?

[Bill Burton]

Hello,

Myself and two other people have been working on reimplementing a
static site in nanoc. We're currently working in a shared virtual
host on our server on several shared git workspaces.

Next we need to do some integration testing combining the nanoc output
with that of the original static site content that has not been
converted yet.

Finally, we need to compile and integrate the site for the production release.

Any suggestions on how to deal with these multiple configurations,
development/sandbox, Q&A/staging and live/production?

Right now the only solution is to put config.yaml in the .gitignore
file which is not ideal as the file is mostly he same for all our
environments. The main differences between environments is the
base_url and images_url settings.

Maybe having nanoc load a config_local.rb or config_local_yaml would
allow overriding settings in config.yaml?

If a configuration override is a Ruby file, then at least in our case,
it could automatically determine the appropriate settings based on the
current directory and hostname.

Thanks,
-Bill

[Denis Defreyne]

Hi,

I don't have a lot of experience with managing multiple environments
with nanoc. There is only a single place where I have some support for
that and that is the deployment rake task, which is described at
<http://nanoc.stoneship.org/docs/6-guides/#deploying-sites>. I have
used multiple configurations quite often for the Myst Online site;
there was a separate "staging" site to where I'd publish the site to
ensure that everything was okay before pushing it to the live server,
and to allow other people to review the site before going live.

The Myst Online site also had a base_url and an assets_url setting,
and I had to change these before compiling and publishing, depending
on where I'd be publishing to. This was indeed a bit of a pain, and it
occasionally went wrong... What you could do, is edit
@config[:base_url] and @config[:assets_url] in the preprocess block
and let those two attributes depend on the environment, e.g.
ENV['nanoc_env']. It's still not a great solution, but it should make
compilation a bit more automated.

If you have suggestions for support for multiple environments in
nanoc, be sure to let me know. Share your ideas!

Denis

> --
> You received this message because you are subscribed to the nanoc discusssion group.
>
> To post to this group, send email to na...@googlegroups.com
> To unsubscribe from this group, send email to nanoc+un...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/nanoc
>

[Bill Burton]

Hello Denis,

Thanks for the tips on dynamically updating the configuration in the preprocess block. 

On Tue, Nov 30, 2010 at 4:10 AM, Denis Defreyne <denis.d...@stoneship.org> wrote:

Hi,

I don't have a lot of experience with managing multiple environments
with nanoc. There is only a single place where I have some support for
that and that is the deployment rake task, which is described at
<http://nanoc.stoneship.org/docs/6-guides/#deploying-sites>. I have
used multiple configurations quite often for the Myst Online site;
there was a separate "staging" site to where I'd publish the site to
ensure that everything was okay before pushing it to the live server,
and to allow other people to review the site before going live.

I pulled your config.yaml from the Myst Online site and it's exactly the same issue we're having with the base_url and assets URL's being different depending on whether compiling locally, to staging or to the live site.
 

The Myst Online site also had a base_url and an assets_url setting,
and I had to change these before compiling and publishing, depending
on where I'd be publishing to. This was indeed a bit of a pain, and it
occasionally went wrong... What you could do, is edit
@config[:base_url] and @config[:assets_url] in the preprocess block
and let those two attributes depend on the environment, e.g.
ENV['nanoc_env']. It's still not a great solution, but it should make
compilation a bit more automated.

That looks like it will work for the nanoc build process but I also need something that can be imported into my Rakefile.

If you have suggestions for support for multiple environments in
nanoc, be sure to let me know. Share your ideas!

One possibility may be to support an environments option in config.yaml like this:

environments:
  production:
    base_url: http://www.example.com
    assets_url: http://assets.example.com
  staging:
    base_url: http://staging.example.com
    assets_url: http://staging.example.com/assets
  development:
    base_url: http://localhost:3000
    assets_url: http://localhost:3000/assets

The selection of the environment would be by setting an environment variable NANOC_ENV or with a new command option -e.  If no environment is specified, development would be used if available.

Best regards,
-Bill

[Damien Pollet]

On Thu, Dec 2, 2010 at 09:29, Bill Burton <bbu...@mail.com> wrote:
> One possibility may be to support an environments option in config.yaml like
> this:

Looks similar to / mergeable with the deployment options, no?

--
Damien Pollet
type less, do more [ | ] http://people.untyped.org/damien.pollet

[Bill Burton]

Hello,

On Thu, Dec 2, 2010 at 4:45 AM, Damien Pollet <damien...@gmail.com> wrote:

On Thu, Dec 2, 2010 at 09:29, Bill Burton <bbu...@mail.com> wrote:
> One possibility may be to support an environments option in config.yaml like
> this:

Looks similar to / mergeable with the deployment options, no?

Yes, it's similar in structure and merging the two would make sense.  For instance:

environments:
  production:
    base_url: http://www.example.com

    deploy:
      dst: "/var/www/default"
      options: [ ... ]
  staging:
    ...

Based on your observation, I started to look around the code that implements the rake deploy:rsync task.  There's useful logic in the Rsync class in Nanoc3::Extra::Deployers that could be repurposed for a generalized Rakefile configuration loader.

-Bill

[Marko]

Hi all,

This is exactly what I've been looking for. Has anyone by chance implemented a feature for configuration management in different environments? Googling gave me just this thread and I'm not familiar enough with the inner workings of nanoc to implement one myself.

Marko

[Denis Defreyne]

Hi,

There is no proper implementation for multiple environment yet, even though I do believe it would be useful.

There are a few important complications to take into account when implementing this. The one that jumps to mind right away is that nanoc uses certain caches to store compilation information so that compilation happens faster. These caches will need to be aware of different environments: cached content from environment X should not be reused in environment Y.

In addition to that, the deploy command may need to be updated to take environments into account as well, so that you don’t accidentally deploy a site compiled as “staging” to a location marked as “production”.

One easy but ugly way to achieve multiple environments is to use a NANOC_ENV variabl, then access it using ENV['NANOC_ENV'] and do conditional stuff based on that. During compilation you’d set the env var on the commandline, e.g. `nanoc co NANOC_ENV=production`. You should remove the tmp directory before compiling your site with a different environment though, because you’ll get incorrect results otherwise.

Denis

> --
> You received this message because you are subscribed to the nanoc discusssion group.
>
> To post to this group, send email to na...@googlegroups.com
> To unsubscribe from this group, send email to
> nanoc+un...@googlegroups.com
> For more options, visit this group at

> http://groups.google.com/group/nanoc?hl=en

[Maik Kempe]

Hi,

maybe you can look at https://github.com/mkempe/nanoczzle, my personal
template based on nanoc3, which have simple environment support.

Or here are the steps for simple multiple environment support:

1. Add this to your config.yaml and set the assets_url in each
environment:

environment: development

development: &defaults
assets_url: /

test:
<<: *defaults
assets_url: http://www.foobar.com/

production:
<<: *defaults
assets_url: http://www.barfoo.com/

2. Add the following method to your helper.rb:

def url_for(item)
if item
environment = @site.config[:environment].to_sym # Get current
environment as symbol

return @site.config[environment] && @site.config[environment]
[:assets_url] ? File.join(@site.config[environment][:assets_url],
item) : item
end
end

3. Use the following helper in your templates / partials:

url_for('/foo.css')
url_for('/bar.css')

4. Change the environment in your config.yaml before you compile.

Done.

[Denis Defreyne]

Hi,

This approach will work because changing the config.yaml file causes all items to be recompiled. If you instead rely on an external source to determine the environment (e.g. an environment variable) this will not work, as nanoc will believe nothing has changed. In this case, you will need to manually recompile (remove tmp/ and compile).

Cheers,

Denis

[jmf...@gmail.com]

Hi Denis,

We have been using nanoc for years for a very large site and are compiling to multiple environments using an environment variable as suggested. Aside from this being not very elegant, the main problem is that the cache in the tmp directory gets invalidated each time we switch environments and so compilation is taking a long time. We have extended nanoc quite heavily with scripts that pipe output through a spell checker, an HTML5 validator, curly quote fixer and more, and a full recompile means those all need to run again.

I'm looking at improving this by symlinking the tmp directory and output directory based on the environment variable, but it would be great if we could get full support for environments by having a cache that is separate for each environment and does not need to be invalidated each time.

Thanks for considering this.
-j

[Bill Burton]

Hello,

I guess having started this thread over four years ago, I should respond :) .

What I ended up doing to largely solve the problem is to use a staging folder to build the site then push it to github.com. In another folder for the live site, I then pull from github.com, recompile the site and the deploy with rsync. The config.yaml is slightly different between the staging and live folders and right now, it's not checked into Git which I don't like.

In any case, this approach eliminates any issues with the tmp/ and output/ folders since the site is built in separate folders for each configuration.

So I really think that attempting to build for multiple environments in a single folder is not a good idea--especially when it's so easy to use a separate folder. You don't have to use github.com to sync, you could create your own bare Git repository and sync with that or use rsync, etc.

It would still be ideal if nanoc could support multiple environments so it's possible to have a single configuration file or files that work in all the environments. In Rails, the YAML files are pre-processed by Erb. Even with that support in nanoc, it would be enough to implement conditional logic in nanoc.yaml without any special support for different environments in nanoc core.

With multiple environment configurations, aside from changing a few configuration items is the desire to pass the environment setting to Compass/Sass so when compiling nanoc production, it would pass that through to the Compass configuration which already has support for production/development, etc. Right now, I run a separate compass compile step for production which compresses the css or for development which generates Source maps and nested css.

Hope this helps,

-Bill

You received this message because you are subscribed to the Google Groups "nanoc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nanoc+un...@googlegroups.com.
To post to this group, send email to na...@googlegroups.com.
Visit this group at http://groups.google.com/group/nanoc.
For more options, visit https://groups.google.com/d/optout.

[jmf...@gmail.com]

@Bill, That's an interesting approach, I may try to replicate that, thanks for sharing. We also build to two different directories but with a bit of a hack: the nanoc Rules file allows you to write to a relative path, so it writes to something like output/../output_production/... based on the environment variable which works but does not solve the issue with invalidating the tmp directory.

It seems that by adding two thing to nanoc we would have a simple fix:
- Add a --config option to nanoc to specify a non-default config file
- Add a new configuration option in the yaml file to specify the location of the tmp directory

I took a brief look at the code and it looks well organised enough that someone familiar with the code could do that in 30 minutes, but perhaps I'm overlooking something

@Dennis would this work? is this something that would be considered for inclusion?

A more elaborate fix would be to also allow you to specify a different Rules file in the config file so that you can apply different filters (Sass compression) but that works well enough with an environment variable.

[Bill Burton]

Hello Jim,

On Mon, 29 Jun 2015 at 14:06 <jmf...@gmail.com> wrote:

@Bill, That's an interesting approach, I may try to replicate that, thanks for sharing. We also build to two different directories but with a bit of a hack: the nanoc Rules file allows you to write to a relative path, so it writes to something like  output/../output_production/... based on the environment variable which works but does not solve the issue with invalidating the tmp directory.

It seems that by adding two thing to nanoc we would have a simple fix:
- Add a --config option to nanoc to specify a non-default config file
- Add a new configuration option in the yaml file to specify the location of the tmp directory

If you use Sass, a .sass-cache folder may be created. Who knows what else needs to be redirected. It could be some filter needs a work area and will create a temporary folder.

If you really feel you need to share the same content folder for different environmental builds, is to have separate folders for each environment build and then symlink the content folder so it's shared among all the environments.

For instance:

  site/

    content/

    staging/

      content -> ../content (symlink)

      Rules, nanoc.yaml, config.rb, etc.

    production/

      content -> ../content (symlink)

      Rules, nanoc,yaml, config.rb, etc.

You don't necessarily have to follow a hierarchy like this. I've not tried this approach with symlinking the content folder, though.

Even if environment variables were supported, you have to be very careful if you build in separate folders to use the correct setting if you build different environments from the same shell prompt.

I took a brief look at the code and it looks well organised enough that someone familiar with the code could do that in 30 minutes, but perhaps I'm overlooking something

@Dennis would this work? is this something that would be considered for inclusion?

A more elaborate fix would be to also allow you to specify a different Rules file in the config file so that you can apply different filters (Sass compression) but that works well enough with an environment variable.

The Rules file is Ruby code so it should be possible to conditionalize most any part of it if required. In contrast, the nanoc.yaml is static.

-Bill

[jmf...@gmail.com]

Yeah, it's a bit tricky, but we wrap the nanoc commands in a rake script that sets up the environment variables and does a few checks to make sure everything is set correctly.

The YAML file is static... it would probably be a one-line addition to nanoc to simply pass the YAML file through Erb like Rails does, that would do away with the need for separate config files.
-jan

[jmf...@gmail.com]

I actually managed to get what I wanted with a tiny monkey patch to nanoc that makes sure the tmp directory does not get invalidated on each run:

Module Nanoc;class Store
def initialize(filename,version)
@filename= filename+ENV["NANOC_PRODUCTION"].to_s
@version=version
end;end

Compiles are under 3 seconds now :-)

[Denis Defreyne]

Supporting multiple environments is definitely something I’d like nanoc to support, either with explicit support, or by providing the basic tools to make a do-it-yourself implementation easy.

I’ve created a ticket for this: https://github.com/nanoc/nanoc/issues/676.

Denis Defreyne
denis.d...@stoneship.org

[jmf...@gmail.com]

On Saturday, July 11, 2015 at 10:57:22 AM UTC+2, Denis Defreyne wrote:
> Supporting multiple environments is definitely something I’d like nanoc to support, either with explicit support, or by providing the basic tools to make a do-it-yourself implementation easy.
>
> I’ve created a ticket for this: https://github.com/nanoc/nanoc/issues/676.
>
> Denis Defreyne

Great, let's continue the discussion there