corrupted size vs. prev_size

[Carlo Giesa]

Hi there!

Since Gaffer 1.6.4.0 (well, we didn't check any 1.6.3.x), we get crashes after Arnold renders have been finished. The renders are done and written to disk, but then, we get following message:

corrupted size vs. prev_size

It seems like this is linked to the LD_PRELOAD update. Is there anything we can do about this?

Thanks,

Carlo

[John Haddon]

Hi Carlo,

I definitely hope something can be done! I think we need to start by pinpointing the problem more precisely. I assume you're on Linux and are rendering with a Render node in the usual way? Via any particular dispatcher? What was the last version that didn't have this problem? 

The change we made in 1.6.4.0 was that we stopped `LD_PRELOAD` changes needed for Gaffer/Arnold from propagating to subprocesses. But if you're rendering in the usual way, then I don't think any subprocesses are involved, unless Arnold launches something.

To confirm that the problem is caused by the change to LD_PRELOAD handling, you could try removing this section from the wrapper in your Gaffer 1.6.4.0 install : https://github.com/GafferHQ/gaffer/blob/main/bin/__gaffer.py#L53-L63.

Once we've got a bit more info, we can start thinking about how to deal with this...

Cheers...

John

--
You received this message because you are subscribed to the Google Groups "gaffer-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gaffer-dev+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/gaffer-dev/f5d9602d-c359-473c-b743-42125bcd4a3en%40googlegroups.com.

[Carlo Giesa]

Hey John!

Thanks for your quick reply.

I continue to search on that matter. It looks like this happens through our tools. When rendering with Arnold, only using gaffer of-the-shelf tools, I don't get the crash.

And when I remove the lines that you are pointing to, the crash disappears.

Do you think it would be possible to add another environment variable to control if we want to clear LD_PRELOAD or not?

Greets,

Carlo

To view this discussion visit https://groups.google.com/d/msgid/gaffer-dev/CAB8pVgJeYAip-XfEea9kxuHZ_x-jro0tUyP0u5EhPv%3DnyXyKyA%40mail.gmail.com.

[Carlo Giesa]

The difference with our inhouse tools is that we don't use 'gaffer execute' but 'gaffer env python /path/to/script.py' and we basically reproduce a very similar code than what can be found in the 'gaffer execute' code.

[John Haddon]

Hi Carlo,

The difference with our inhouse tools is that we don't use 'gaffer execute' but 'gaffer env python /path/to/script.py' 

Good hunting - seems like we're getting somewhere. The underlying issue here is that if `libstdc++.so` isn't loaded before `libai.so`, then some (but not all) users of global `operator new` and `operator delete` resolve to symbols exposed by `libai.so` (which internally has its own memory allocator). That causes crashes when things allocated by one memory allocator get freed by another.

Most DCCs (and `kick` itself) link to `libstdc++.so` directly so this isn't an issue for them. But Python is written in pure C, so doesn't link to it (and Gaffer is just a Python program). So we LD_PRELOAD it instead. And now that we are cleaning up this LD_PRELOAD from the environment for subprocesses, your Python subprocess is exposed to the same problem.

Do you think it would be possible to add another environment variable to control if we want to clear LD_PRELOAD or not?

Ugh! I felt bad enough adding the LD_PRELOAD in the first place, and worse again when adding the temporary environment variables used to do the cleanup. I'd really like to avoid another variable on top of all that. I think such a variable would likely only work in the short term anyway. For various reasons, we're becoming increasingly likely to introduce a custom Gaffer executable instead of just using Python as our launcher. That would be implemented in C++, so would link directly to `libstdc++.so`, and we'd remove the LD_PRELOAD hack entirely at that point. Which would put you back to square one with your own Python process.

Here are some alternatives you might be able to consider :

• If you have a Gaffer wrapper, put the same LD_PRELOAD in that. Gaffer will honour any settings from the environment it was launched in, passing them to the subprocess.
• Use `gaffer python` instead of `gaffer env python`, since that runs your script in the main Gaffer process, not a subprocess.
• Use `gaffer env LD_PRELOAD=...` to reintroduce the preload just before running python.
• Refactor your custom script into a Gaffer app (like `execute`), so it can be launched by `gaffer yourApp` and therefore not be run in a subprocess.
• Use `ctypes` to load `libstdc++.so` at the very top of your python script, before Arnold gets imported.

Hopefully one of those works for you. If not, let us know and we can consider further...

Cheers...

John

 

[Carlo Giesa]

Hi there!

Use `gaffer python` instead of `gaffer env python`, since that runs your script in the main Gaffer process, not a subprocess.

That did the trick. How wonderful if things can get solved as easily as this. What exactly would be the purpose of using 'gaffer env python' instead of 'gaffer python'? Si this for legacy or are there more reasons for its existence?

Greets,

Carlo

--

You received this message because you are subscribed to the Google Groups "gaffer-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gaffer-dev+...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/gaffer-dev/CAB8pVgJSjyDeP_bmyEjsWJu7JPtYth_s4Z0ZRF5Qr8-eA_xpgw%40mail.gmail.com.

[John Haddon]

Glad that worked for you!

`gaffer python` predates `gaffer env`, and to be honest we kindof thought it had been superceded by it. But this new use case and a few other benefits make me think `gaffer python` will stay around for a while.

Cheers...

John

To view this discussion visit https://groups.google.com/d/msgid/gaffer-dev/CANHZvCO%2BRnCYZ_ALAn%3DLtBtPP7LR%2BuQLw6eHBYiNRRvyR2VeFw%40mail.gmail.com.