Thoughts on layering of multiple config files

[natha...@gmail.com]

Hey all,

I hope I'm not spawning a duplicate discussion thread; I did some searching around, but couldn't find anything.

I'm wondering if there has been any discussion about allowing multiple config files to be used to build a single runtime configuration by layering multiple (possibly sparse) definitions on disk. Similar to the PATH environment variable, the final configuration would be built from the "bottom" of the OCIO variable's path list up, allowing files of higher precedence to replace existing definitions within the configuration (to facilitate overrides), as well as add items to it. One goal of this would be to make it easy to propagate changes to "static" pieces of a core OCIO configuration across many shows.

How are people currently handling the possibility of changes to what might be considered the "core" of your OCIO configuration? One option seems to be to have the build script for each show's configuration load a "baseline" config and then amend it or replace certain items (assuming this is possible)...

Thanks for any thoughts,

-Nathan

[Jeremy Selan]

Hi.

Unfortunately the ability to layer ocio configurations isnt implemented yet, though we do consider it a really good idea.  See this existing issue for more information, or to add your thoughts:

https://github.com/imageworks/OpenColorIO/issues/282

In the meantime, we handle this at SPI using python scripts to build configurations for each show. The code is generally similar to this...

https://github.com/imageworks/OpenColorIO-Configs/blob/master/nuke-default/make.py

Not quite as elegant as being dynamically read on the fly, but it has suited our needs to a high enough degree that this has been a low priority for us in the interim.

-- Jeremy

--
You received this message because you are subscribed to the Google Groups "OpenColorIO Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ocio-dev+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Rob Molholm]

Hey folks,

Although implementation of issue #282 below is still ideal, I have been testing a combination of FUSE and OCIO, using FUSE as a way to abstract database queries, presenting results as "virtual" OCIO configurations to the calling application (using the filesystem hierarchy for baked context).   This does have its limitations, but it is one way to centrally manage several configurations.  FUSE also works across multiple platforms, which is a bonus.

In production we're pretty much doing as mentioned-- we have a baseline OCIO configuration, which is carried over and modified by a script for each project, and also baked for contexts within a project where the OCIO contexts do not work

...and speaking of including/merging configurations- has it ever been floated to add an OCIO config file as one of the supported file types of OCIOFileTransform?    there'd have to be an additional argument to specify a specific color space (or look) out of the included file, but this could be a way of bringing in known complex/grouped transforms, rather than relying on the basics in a single file.  The source path of the file transform would be subject to the current context, so you definitely could get crazy with this.  Not sure how dependancies from the included color space would be resolved either...

-rob

[Malcolm Humphreys]

I was hoping we could use a combination of anchors and references (http://en.wikipedia.org/wiki/YAML#References) with some kind of special '#include some_other_$(SHOW).ocio' which would be resolved before it gets to the yaml parser inside OCIO.

Support for anchors and references would also allow to reduce a lot of duplication in current profiles while also opening up the possibility of using data from profile segments that get #included.

.malcolm

[Nathan Rusch]

Thanks for the information everyone.

It sounds like the general thoughts in this direction so far are leaning toward the idea of nesting or referencing configurations within each other. While I think this could certainly be useful, it really seems like a layering system would be far simpler to implement, as well as to actually make use of.

My basic thought is that the different configuration files would be completely blind to each other, and it looks like Config::Impl::load() could be safely called once for each file found on the environment variable. With this approach, the burden of making sure you aren't adding references to undefined transforms or colorspaces could fall on the end user (at least initially). Among other things, this would allow for, say, a single application to easily override the definitions for certain transforms, looks, etc. without requiring what would otherwise be a carbon copy of the original config file.

What are the general reactions to configuration layering vs. nesting, or thoughts on the implementation side?

-Nathan

P.S. Also, just for reference, Hugh pointed me to a topic he started in 2011 that pertains to the nesting/cross-referencing of configurations (I wasn't using the right search terms, so I didn't see it): https://groups.google.com/forum/#!topic/ocio-dev/okBHXi1ibnM.

[dbr/Ben]

There is a ticket for this, https://github.com/imageworks/OpenColorIO/issues/282

Jeremy's reply to https://groups.google.com/forum/#!topic/ocio-dev/okBHXi1ibnM sums up most of my concerns about this feature nicely..

Merging two arbitrary YAML configs seems like a direction in which madness lies.. There's lots of cases where the behaviour would be non-obvious, e.g:

* parent defines and uses a 'linear' space, child also defines a 'linear' space with, say, different allocation - what happens?

* Does a file-reference in the parent search the child search_path?

* If there's a base config, can I not inherit a specific colorspace?

* ..and a bunch more I haven't fully thought through (like "would you need to control the order in which views are merged?", "in the child config, could you remove a colorspace from the parent?")

Limiting the scope a bit might make things clearer, e.g something like

colorspaces:

  - !<IncludeColorspaces> {src: "/path/to/shared/base.ocio", include: [srgb, "*log"], exclude: ['badlog']}

  - !<Colorspace>

     name: ...

Where this would just include the colorspace definitions, but ignore anything like the search_path from the included file. If the included config requires LUT's in a certain location, this path would be added to "search_path" as usual.

That said, I don't really understand the benefit of this.. While the current "one config.ocio per project" setup involves some duplication of colorspace definitions etc, I don't see this as a downside.. Either a colorspace never changes (in which case, copying-and-pasting is little hassle), or the colorspace needs changed (in which case, it's useful being able to update new projects without fear of breaking current/finished projects)

It'd be good to have a few clear explanations of workflows where the nested configs would be beneficial (beneficial over alternative approaches like copy-pasting-and-modifying, or Python-generated configs)

- Ben

[Nathan Rusch]

Hi Ben,

I did see your responses to Hugh's thread after he pointed me to it. However, once again, the discussion seems to be centered around the idea of nesting or referencing config files, whereas I'm mainly interested in being able to layer config files together for very specific overrides. I'll try to sketch my thoughts out a little more clearly:

• Base configuration lives at `/path/to/base.config` (this could be facility-wide or show-specific; that part isn't important).
• Set up OCIO environment var in the same way you might set up $PATH, except with OCIO configs
• export OCIO=/path/to/sparse/override.config:/path/to/base.config
• OCIO creates a Config object from the environment by doing the following:
  

1. Load the configuration file found at the *back* of the $OCIO variable (in the above case, the "base config")
2. Working from back to front, load each config definition onto the Config instance that was created in step 1.
3. Only sanity-check the resulting Config instance after all paths have been loaded.

In this example, the file `/path/to/sparse/override.config` would be able to, say, replace a single ColorSpace definition with a more complex set of transforms or add a Look by *only* containing the desired additional or override definitions (and, say, the ocio_profile_version key). If the sparse file defined a ColorSpace with a name that matched the name of a ColorSpace in the "base config," the sparse file's definition would replace it purely based on the order in which the config files appear on the $OCIO variable.

This would allow for overrides without requiring any preprocessing of the YAML data (to, say, resolve includes), any syntactical changes to the configuration files, or any alterations to any *existing* configurations. We use this same kind of thing to manage stored preferences/recent settings for pipeline operators at different contextual levels (show, shot, element, action, user). It allows each type of preference to be as sparse as possible to avoid writing out the same data many times, and for a given application or scenario, the order in which these contextual preferences "squash" each other can change (in the case that duplicate preference keys are possible).

The main scenario I can see for deploying this type of config "patching" is when the crazy requests start rolling in from production near the end of a show, when it is often critical to be nimble. You are correct that even this type of extension would give anyone more wiggle room to break things, and that it wouldn't provide a way to remove things from the final Config. However, I don't really see this layering idea being the standard way of doing things, but simply an added dimension of flexibility.

There is currently nothing stopping anyone from manually breaking an existing configuration with a text editor, or writing a malformed definition from scratch, and it isn't OCIO's responsibility to prevent them from doing this. In that same spirit, it wouldn't be OCIO's responsibility to prevent someone from trying to combine two configuration files that don't result in a valid Config object.

Hope this helps clarify my thoughts a little more, and how they differ from the idea of referencing or including configuration files between each other.

-Nathan

[Mark Boorer]

Hi,

Apologies for bumping an old discussion, but I'm in looking for a similar set of features to the ones Nathan described, and as the discussion seemed to have fizzled out, I wanted to show my interest.

I would also like to have the OCIO env var behave like the $PATH variable, and be able to override parts of the resolved OCIO::Config in further, subsequent config.ocio files.

As long as each config file was also able to stand alone, the only real roadblock seems to be that the Config object currently holds a context, which looks like it's only used by the FileTransform.

Is it possible to have the FileTransformation instances contain their own context? So if they are overridden later when the next config is loaded, then the new ones will work with the new paths, and the old ones would continue to work with their old paths.

I'm quite interested in getting this functionality, specifically to override roles on a show / shot level, and would be happy to implement the required changes.

Cheers,
Mark