Gaffer crashes with Yeti

[Carlo Giesa]

Hi there!

We are facing unexpected crashes of Gaffer when rendering Yeti. It happens very probably at shutdown since the rendered image exists and looks like expected. Rendering without Yeti works like a charm.

We are running almost the same config on this project (Yeti 4.2.11, Arnold 7.2.4.1) as we did already on another project where we did not encounter such crashes. The only difference is that we switched Gaffer from 1.3.10.0 to 1.3.11.0. And we tested all available versions > 1.3.11.0 with the same negative result.

I double checked the release notes and the diff between 1.3.10.0 and 1.3.11.0 and I can't see anything that could explain this. Tests on Rocky 9 (which is not yet in production) were not giving any better result.

If we rerun the render again and again, at some point, it does not crash. So, there seems to be some random part in that problem, although, crashes happen most of the time.

We did check with 'GAFFER_DEBUG=1'. In that case, the crash seems to disappear magically. We tried with 'LD_PRELOAD=/usr/lib64/libSegFault.so', but this just printed out something completely weird, containing things that look like memory addresses, but nothing really understandable and helpful.

We could not really test with Gaffer 1.4.x.x since the test scene did not work at all. I guess that there are some breaking changes that would need to be checked first. Trying on Rocky 9 didn't have any positive impact.

If you might have anything that could point us into some direction, it would be awesome!

Greets,
Carlo

[Daniel Dresser]

That is very odd - I just skimmed the actual commits between 1.3.10.0 and 1.3.11.0 and I'm not seeing anything suspicious.

The first thing I can think of to check is to set your render node Mode to "Scene Description" and set the `File Name` to something, to produce an Arnold scene description file you can render with Arnold's `kick` command. Given the dependency on Gaffer version, I'm guessing that probably isn't going to crash, but it would make everything much easier to figure out if it did, so it's worth a shot.

Other than that, I'll try to think of anything else that might be useful ... a lot of the team is out until Tuesday, so we might have some more ideas then.

-Daniel

[John Haddon]

On Thu, May 30, 2024 at 5:30 PM 'Carlo Giesa' via gaffer-dev <gaffe...@googlegroups.com> wrote:

We are running almost the same config on this project (Yeti 4.2.11, Arnold 7.2.4.1) as we did already on another project where we did not encounter such crashes. The only difference is that we switched Gaffer from 1.3.10.0 to 1.3.11.0. And we tested all available versions > 1.3.11.0 with the same negative result.

Dumb question, but did you test 1.3.10.0 on this new show as well? To rule out some other change being the cause?

 

If you might have anything that could point us into some direction, it would be awesome!

My mind immediately goes to some sort of conflict between Yeti's dependencies and Gaffer's. Is that possible? Does Yeti link to much in the way of shared libraries?

[Carlo Giesa]

Le mardi 4 juin 2024 à 16:53:51 UTC+2, John Haddon a écrit :

On Thu, May 30, 2024 at 5:30 PM 'Carlo Giesa' via gaffer-dev <gaffe...@googlegroups.com> wrote:

We are running almost the same config on this project (Yeti 4.2.11, Arnold 7.2.4.1) as we did already on another project where we did not encounter such crashes. The only difference is that we switched Gaffer from 1.3.10.0 to 1.3.11.0. And we tested all available versions > 1.3.11.0 with the same negative result.

Dumb question, but did you test 1.3.10.0 on this new show as well? To rule out some other change being the cause?

Yes, indeed.

 

If you might have anything that could point us into some direction, it would be awesome!

My mind immediately goes to some sort of conflict between Yeti's dependencies and Gaffer's. Is that possible? Does Yeti link to much in the way of shared libraries?

The only .so files that I could find in the yeti installation are following:

I don´t know if that answers your question.

The thing that I can remember is that I always had to juggle with different versions of Arnold and Yeti to have a "working" combination. In the beginning of a recent show, we tried to use the latest versions at that moment which did crash as well and moving back to a slightly older one for Arnold and Yeti "fixed" the issue.

But here, I tried a bunch of combinations of versions, and from Gaffer 1.3.11.0 onwards, I can't find the magic working combination :(

Our current work around is to ignore the segmentation faults for render jobs including yeti. That is of course a pretty dirty solution...

Greets,
Carlo