Gaffer 1.6 - merge exr issue

[s.des...@gmail.com]

Hello,

I'm testing the new features of Gaffer 1.6, especially the Merge EXR feature.
I’ve created a simple setup where I define some default outputs that all share the same file path. This setup works fine, and I can see all the layers in my EXR.

However, when I try to customize my outputs by changing the filter or preserve layer name parameters, I get the following errors:

ERROR   [Render] Render.task : Mismatch in combined output driver for file name ".../renders/gaffer_simple_setup_v001/layer01/multiExr.0001.exr" : value of parameter "filter"
ERROR   [Render] gaffer execute : executing Render : See previous message for details
ERROR   [Render] Execution failed for frame 1

It seems that all the outputs must share exactly the same parameters to avoid errors.

This is problematic because I want to include denoised AOVs in my EXR using a variance filter.

Am I missing someting or is it a limitation ?

Thx,

Seb

[John Haddon]

Hi Seb,

When multiple outputs are all targeting the same driver (filename), we do some checks to verify that the outputs don't have conflicting parameters. Some output parameters are actually driver settings, so if two outputs have different opinions about those, they can't both get what they want. The `preserve_layer_name` parameter is an example of this - if you run `gaffer env kick -info driver_exr` you'll see that it is a parameter of the driver itself, so all outputs need the same value.

For the `filter` parameter this check doesn't seem appropriate though, because it's not a driver parameter and can be varied from output to output. So that part at least looks like a Gaffer bug to me.

Daniel might know more, as he was the one who implemented the feature. He's out this week, but perhaps you could open a GitHub issue for us to track, and he can take a look when he's back?

Cheers...

John

--
You received this message because you are subscribed to the Google Groups "gaffer-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gaffer-dev+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/gaffer-dev/deb47a0e-b5e7-4682-81dc-afd987038509n%40googlegroups.com.

[s.des...@gmail.com]

Hello John,

Thx for the reply. I open an issue on GitHub.

Seb

[s.des...@gmail.com]

Hello John,

I’ve tested the new Gaffer 1.6.1.0 and the EXR merge seems to work fine. I can merge multiple AOVs and denoise them with the Arnold Noice command line.

There’s only one thing I can’t get to work as expected. If I enable layerPerLightGroup in my beauty AOV to generate the light groups, the RGBA channel doesn’t appear in the EXR file.
When I check the file in Nuke, I can see the light group channels, but the rgb and rgba channels are empty.

I found a workaround by creating a custom output with Arnold light path expressions, which makes the rgb and rgba channels visible. However, this breaks the EXR channel metadata and the denoiser no longer works.

Do you have any idea how I can fix this, or is it a bug?

Thanks,

Seb

[John Haddon]

Hi Seb,

Unless I've missed something, you can get what you want by adding a second `rgba` output with `layerPerLightGroup` turned off, as in the screenshot below :

Does that work for you?

Cheers...

John

To view this discussion visit https://groups.google.com/d/msgid/gaffer-dev/83695d6e-6bff-443b-a75e-21e1ea534692n%40googlegroups.com.

[s.des...@gmail.com]

Hello John,

It was simple as that... I had indeed created a layerPerLightGroup attribute for the lightgroup output, but I hadn’t created it on the beauty, thinking it wasn’t necessary.

Now it works perfectly !

Just one last question:

I guess the outputs are written and merged into a single-part EXR by default, otherwise Arnold Noice wouldn’t work.
But is there an option in Gaffer to merge the outputs into a multi-part EXR? In Maya, there is an option to render the EXR as multi-part.

If I use the Arnold denoiser, I have to write another EXR file, and then I can choose multi-part EXR. But if I don’t need the Arnold denoiser, it would be nice to render directly to multi-part EXR to avoid the extra post-process step.

Thx,

Seb

[John Haddon]

On Mon, Sep 29, 2025 at 9:48 AM s.des...@gmail.com <s.des...@gmail.com> wrote:

It was simple as that... I had indeed created a layerPerLightGroup attribute for the lightgroup output, but I hadn’t created it on the beauty, thinking it wasn’t necessary.

Now it works perfectly !

I'm not sure the `layerPerLightGroup` is actually necessary on the beauty - I think the important thing is that `data` is `rgba` , whereas before you had mentioned you'd used an LPE of some sort instead. In any case, I'm glad it's working!

But is there an option in Gaffer to merge the outputs into a multi-part EXR? In Maya, there is an option to render the EXR as multi-part.

Arnold's `driver_exr` has a boolean `multipart` parameter (that defaults off). So if you add a `multipart` parameter to all your outputs and turn it on, then I would expect Arnold to write multipart. I haven't tested though - let us know how you get on...

Cheers...

John

[s.des...@gmail.com]

Hello John,

Thx, I’ll give multi-part a try to see if it works.

I’m also trying to see if I can get the Arnold Noice denoiser to work. It works fine as long as I don’t use the temporal flags, but when I do, I get this warnings in the logs:

- Frame numbering from metadata is not consistent, using default indices.

I checked, and it seems Noice uses the  "EXR/arnold/frame" metadata for temporal denoising. When I inspect my EXR metadata, the "EXR/arnold/frame" value stays at 0 across all frames.

Is there a way to fix this? I checked in Maya, and the "EXR/arnold/frame" metadata is dynamic there, I can see the correct frame number.

Thx,

Seb

[John Haddon]

Is there a way to fix this? I checked in Maya, and the "EXR/arnold/frame" metadata is dynamic there, I can see the correct frame number.

Sounds like this is something we should fix in Gaffer, but I think you can work around it for now. If you use a CustomOptions node to create an `ai:frame` float option, and use an expression to set it to the right value, that seems to carry through to the EXR metadata.

Cheers...

John

[s.des...@gmail.com]

Hello John,

Thx for the metadata workaround !

I 've also tested the multipart parameter, and it seems to work.

I just have one remark about it: since the multipart parameter must be the same for all outputs in case of merge exr, wouldn’t it be a better idea if the multipart option lived in the Arnold render options instead of in each output?

Thx,

Seb

[John Haddon]

Thx for the metadata workaround !

No worries - I've made the proper fix now too : see https://github.com/GafferHQ/gaffer/pull/6619. 

I just have one remark about it: since the multipart parameter must be the same for all outputs in case of merge exr, wouldn’t it be a better idea if the multipart option lived in the Arnold render options instead of in each output?

I can see how that would be convenient for the most common use cases, but it would stop someone from making a combination of multi-part and single-part outputs. It's our general policy to expose each renderer's options in as direct/literal a way as possible, and since the `multipart` parameter is on `driver_exr` in Arnold rather than the `options` node, it maps most naturally to output parameters in Gaffer.

[Matthias Bjarnason]

Hey all,

We've just encountered an issue that probably belongs here in this thread. Since it's related to EXR Merges.

We're seeing strange issues with Arnold AOV EXR merges in Gaffer 1.6.2.1 (Arnold 7.4.3.0) when certain AOV/Driver attributes are set in the Outputs node.
It seems that Gaffer/Arnold messes up the order of the channels written into the merged EXR.

We did not spot this earlier because when the resulting EXRs were viewed in Nuke and/or Gaffer's imageReader the result looked fine - both "readers" appear to look past this issue and show you the RGB channel by default. BUT by inspecting the EXRs metadata in Nuke or simply reading the files in RV one can see the that something is not right.
Instead of RGB(A) being the first channel on the EXRs list, the other AOVs get randomly chosen instead (per frame), so during playback ...for each frame, a random AOV is on top (of the channel list) and images flash like crazy.

Here are examples from the EXR data ("oiiotool --info -v") for the first four images in the sequence;

(#1001)
channel list: sheen_direct.R, sheen_direct.G, sheen_direct.B
oiio:subimagename: "sheen_direct"
oiio:subimages: 14


(#1002)
channel list: sheen_direct.R, sheen_direct.G, sheen_direct.B
oiio:subimagename: "sheen_direct"
oiio:subimages: 14


(#1003)
channel list: specular_indirect.R, specular_indirect.G, specular_indirect.B
oiio:subimagename: "specular_indirect"
oiio:subimages: 14


(#1004)
channel list: sss_direct.R, sss_direct.G, sss_direct.B
oiio:subimagename: "sss_direct"
oiio:subimages: 14


At first we thought this was happening ONLY when "layerName" attribute was specifically defined but it seems that this happens in all cases where a "non-default" attribute is applied, meaning attributes not automatically created when hitting the plus sign in the Outputs panel and selecting an AOV.

Similar thing happens when "preserve layer name" is set/enabled ...and then this goes absolutely crazy when "Multipart" is enabled - No matter the combination of enabled attributes, it's never correct.
In order for the Merged AOV output to have the "correct" channel list order, all three attributes (at least) must be disabled i.e. "layerName", "preserve_layer_name" and "multipart".

Example frame of that settings results in: ("oiiotool --info -v")

channel list: R, G, B, A, coat_direct.R, coat_direct.G, coat_direct.B, coat_indirect.R, coat_indirect.G, coat_indirect.B, diffuse_albedo.R, diffuse_albedo.G, diffuse_albedo.B, diffuse_direct.R, diffuse_direct.G, diffuse_direct.B, diffuse_indirect.R, diffuse_indirect.G, diffuse_indirect.B, sheen_direct.R, sheen_direct.G, sheen_direct.B, sheen_indirect.R, sheen_indirect.G, sheen_indirect.B, specular_direct.R, specular_direct.G, specular_direct.B, specular_indirect.R, specular_indirect.G, specular_indirect.B, sss_direct.R, sss_direct.G, sss_direct.B, sss_indirect.R, sss_indirect.G, sss_indirect.B, transmission_direct.R, transmission_direct.G, transmission_direct.B, transmission_indirect.R, transmission_indirect.G, transmission_indirect.B
oiio:subimages: 1

...wastly different to the previous examples - all available channels are listed here as one sub-image in the info (possibly legacy format i.e. not multipart EXR 2+).
This arrangement is "functional" and at least not broken, so we can get our renders through our pipeline but, for obvious reasons, having Multipart EXRs is way more efficient and being able to set "nice-names" for AOVs is always handy as well. This all goes without saying that being rid of the "old" post render merges frees up a good amount of resources from our renderfarm as well.

For comparison here is the EXR info data from a file that was "post merged" using Gaffer's imageWriter (prior to 1.6), set to "Part per Layer":

channel list: R, G, B, A
oiio:subimagename: "rgba"
oiio:subimages: 17


TLDR  ...Has anyone here experienced the same and/or noticed a similar issue when "merging" AOVs to single EXR using Arnold in Gaffer 1.6+? Is Gaffer somehow instructing Arnold on how to arrange those channels or is this completely handled by Arnold (render logs show no hint as to what the order will be)? Can anyone please try to replicate this on their end to see what happens with their merged EXR data?

Any info appreciated - Thanks.

Best,
-M

[dan...@image-engine.com]

If you're curious about debugging this sort of thing, you can use a Render node with Mode set to "Scene Description" to dump an ascii file with everything Gaffer is sending to Arnold.

In this case, it's pretty obvious to see that we're passing the outputs to Arnold in a random order ( more specifically, it's using our fast string comparison which is based on memory address, which is consistent within a process but not between processes ). Whatever the Arnold driver is doing when it combines things, I strongly suspect that the inconsistency is because we don't specify the output list in a consistent order.

We definitely should make this consistent. The simplest way to do that would just be sort the outputs alphabetically based on their names using real string comparisons. That would ensure consistency - which would be my main concern.

You make reference to a "correct" order in your message ... would just having it be consistent be good enough for you, or do you want to be able to specify a particular order somehow? Standard's compliant exr readers don't really care much about what order things are stored in, so I guess I would hope that just consistent might be good enough. If someone has a pipeline where they need the beauty first ... I guess it would be an option to rename their beauty to "AAABeauty". Not pretty, but it would work ( once we fix things to use alphabetical order ).

The alternative would be to add some notion of order to outputs, and some way for users to control that ... adding more interface like could easily get more complicated though, as opposed to just sorting based on alphabetical order, which should be very quick to implement.

-Daniel

[Matthias Bjarnason]

Thanks for the info Daniel - super helpful.

Far from being an expert on any OpenEXR standards (if there are any in relation to this issue) but I would assume the channel order should be set in an alphabetical order, as you suggest, with R,G,B,A always being on the 'top of the stack' (in the most simple terminology).
That is what I was referring to with the term "correct". I might be completely wrong but, from what I can tell, that would be most compatible (or expected input) with other image readers out there.

Now, I'll admit that I've not had the chance to render from other DCCs to see how they handle this thing.

In the example I added here above (legacy - all options off), you can clearly see that this is the case. R,G,B,A come first and then the rest of the AOVs in an alphabetical order - Although only a single subimage (non multipart) it might still be an indication of the 'standard' way of doing this.

Just out of curiosity, what is the mechanic within the imageWriter node? In our "post merge" process method, we're not deliberately feeding separate AOV renders alphabetically into the CollectImages node and yet the output does come out "correct".

I can not immediately see a use-case wherein we would ever want the order to be something specific beyond that rule, or any options being necessary for a custom output - rendering deep data or anything else. Perhaps someone else here might be able to chime in on that.


Thanks again,

Best
- M

[John Haddon]

On Fri, Oct 31, 2025 at 10:06 AM Matthias Bjarnason <matthias....@rvx.is> wrote:

Far from being an expert on any OpenEXR standards (if there are any in relation to this issue) but I would assume the channel order should be set in an alphabetical order, as you suggest, with R,G,B,A always being on the 'top of the stack' (in the most simple terminology).

As far as I know, OpenEXR doesn't say anything about channel or part ordering. Simple example : if you query the channels from a single part EXR file via `exrheader`, they'll actually come out as "B G R", not "R G B"! OpenImageIO resorts the channel names when loading so that they make a bit more sense, but I presume that sorting is done per-part rather than holistically.

That is what I was referring to with the term "correct". I might be completely wrong but, from what I can tell, that would be most compatible (or expected input) with other image readers out there.

 

Yeah, putting RGB in the first part does make sense from a pragmatic standpoint. I think Daniel may just be quibbling about the word "correct" - a truly robust image reader shouldn't really care. 

In the example I added here above (legacy - all options off), you can clearly see that this is the case. R,G,B,A come first and then the rest of the AOVs in an alphabetical order - Although only a single subimage (non multipart) it might still be an indication of the 'standard' way of doing this.

I think `oiiotool` is doing the sorting for you here - `exrheader` would give you a more truthful readout of the order within the file.

 

Just out of curiosity, what is the mechanic within the imageWriter node? In our "post merge" process method, we're not deliberately feeding separate AOV renders alphabetically into the CollectImages node and yet the output does come out "correct".

The ImageWriter sorts before writing. Internally in Gaffer we don't care what the channel order is. But we sort when displaying to the user or writing to file.

 

I can not immediately see a use-case wherein we would ever want the order to be something specific beyond that rule, or any options being necessary for a custom output - rendering deep data or anything else. Perhaps someone else here might be able to chime in on that.

 I've taken a stab at fixing this in https://github.com/GafferHQ/gaffer/pull/6668. It ensures that the RGB/RGBA part comes first, and the other parts come after, sorted alphabetically by layer name.

Cheers...

John

[Matthías Bjarnason]

 I've taken a stab at fixing this in https://github.com/GafferHQ/gaffer/pull/6668. It ensures that the RGB/RGBA part comes first, and the other parts come after, sorted alphabetically by layer name.

Excellent! - Thanks John, looking forward to trying this out once deployed.

I think `oiiotool` is doing the sorting for you here - `exrheader` would give you a more truthful readout of the order within the file.

Sneaky monkeys :)  ... good to know for future detailed inspection of files. 

Fantastic work guys  - thanks again!

Best,

--
Matthías Bjarnason
Creative Director
RVX
Brúarvogur 1-3
104 Reykjavík, Iceland
www.rvx.is

--
You received this message because you are subscribed to a topic in the Google Groups "gaffer-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gaffer-dev/nAnRyHTLCUI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to gaffer-dev+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/gaffer-dev/CAB8pVg%2B_ufv0mrmeBMRdx%2B%2B2NLApnmzq9bCLfxQW5ypzC0AUcg%40mail.gmail.com.