Houdini crashes instantly
Francois Lord
Hi,
We have upgraded to RRv9.1.20 and rez to 3.2.1 over the last few months and we now need to render frames for the first time since April. When we try rendering with Houdini (any method), it crashes instantly. I don't know what else we might have changed. I tried with Houdini 21 and 20.5 which used to work fine.
What could this be?
The command RR uses is something like this:
\\path\to\rez\rez-env.exe houdini-21.0.440-r1 htoa-6.4.4.0-r1 [...more rez packages] -- "hython" "C:/RR_localdata/renderscripts/kso_houdini.py" -scene "//path/to/scene/scene__simpleKarmaH21.hipnc" -renderer "createUSD" -exportmode true -FrStart 1 -FrEnd 1 -FrStep 1 -FPadding 1 -FName "////path/to/scene/geo/scene__simpleKarmaH21.usd_rop1.usd" -FSingleFile True -rop "/stage/usd_rop1" -threads 20 -renderDemo False -avFrameTime 0 -AASamples 1.0 -noFrameLoop True -unlockAssets True -PyModPath C:\RR_localdata\renderscripts
And the log looks like this, it can't find python311.dll:
R 7| (0) Couldn't load -/bin/python311.dll.
R 8|
R 9| Le module spécifié est introuvable.
R 10|
R 11| 13804: Fatal error: Segmentation fault
R 12| -- TRACEBACK BEGIN --
R 13| Traceback from hython 21.0.440 (Compiled on windows-x86_64-cl19.42):
R 14| CURRENT THREAD 19100
R 15| +0x0441094a [PYsetArgvForPythonInitialization] C:\REZ\houdini\21.0.440-r1\95dd\a\payload\bin\libPY.dll
R 16| +0x04410752 [PYmakeSurePythonIsInitialized] C:\REZ\houdini\21.0.440-r1\95dd\a\payload\bin\libPY.dll
R 17| +0x044104f1 [PY_BaseInterpreterAutoLock::PY_BaseInterpreterAutoLock] C:\REZ\houdini\21.0.440-r1\95dd\a\payload\bin\libPY.dll
R 18| +0x140075ad7 C:\REZ\houdini\21.0.440-r1\95dd\a\payload\bin\hython.exe
R 19| +0x140077f2a C:\REZ\houdini\21.0.440-r1\95dd\a\payload\bin\hython.exe
R 20| +0x1400783f0 C:\REZ\houdini\21.0.440-r1\95dd\a\payload\bin\hython.exe
R 21| +0x14014f480 C:\REZ\houdini\21.0.440-r1\95dd\a\payload\bin\hython.exe
R 22| +0x7ffcd3dfe8d7 [BaseThreadInitThunk] C:\WINDOWS\System32\KERNEL32.DLL
R 23| +0x7ffcd55e8d9c [RtlUserThreadStart] C:\WINDOWS\SYSTEM32\ntdll.dll
R 24| OTHER THREAD 7760
R 25| +0x7ffcd5746ee4 [ZwWaitForWorkViaWorkerFactory] C:\WINDOWS\SYSTEM32\ntdll.dll
R 26| +0x7ffcd565e02e [RtlRaiseStatus] C:\WINDOWS\SYSTEM32\ntdll.dll
R 27| +0x7ffcd3dfe8d7 [BaseThreadInitThunk] C:\WINDOWS\System32\KERNEL32.DLL
R 28| +0x7ffcd55e8d9c [RtlUserThreadStart] C:\WINDOWS\SYSTEM32\ntdll.dll
R 29| OTHER THREAD 20612
R 30| +0x7ffcd5746ee4 [ZwWaitForWorkViaWorkerFactory] C:\WINDOWS\SYSTEM32\ntdll.dll
R 31| +0x7ffcd565e02e [RtlRaiseStatus] C:\WINDOWS\SYSTEM32\ntdll.dll
R 32| +0x7ffcd3dfe8d7 [BaseThreadInitThunk] C:\WINDOWS\System32\KERNEL32.DLL
R 33| +0x7ffcd55e8d9c [RtlUserThreadStart] C:\WINDOWS\SYSTEM32\ntdll.dll
R 34| OTHER THREAD 20200
R 35| +0x7ffcd5746ee4 [ZwWaitForWorkViaWorkerFactory] C:\WINDOWS\SYSTEM32\ntdll.dll
R 36| +0x7ffcd565e02e [RtlRaiseStatus] C:\WINDOWS\SYSTEM32\ntdll.dll
R 37| +0x7ffcd3dfe8d7 [BaseThreadInitThunk] C:\WINDOWS\System32\KERNEL32.DLL
R 38| +0x7ffcd55e8d9c [RtlUserThreadStart] C:\WINDOWS\SYSTEM32\ntdll.dll
R 39| -- TRACEBACK END --
R 40|
R 41| >"C:\Program Files\RoyalRender\lc64\rrCheckexitcode.exe" 139 0 0 1
R 42| ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
0| ++ Nov 11. 20:30.50 Render executable closed ++++++++++++++++++++++++++++++++++++++++++++++++++++
1| ++ Royal Render checks the return code of the executable or the OS +++++++++++++++++++++++++++++++
2| ++ Executable returned 139 (0x8b) as exit code for frame 1 (jobthread 0, instance 0). ++++++++++++
3| ++ Error COULD be: Error 139: Le syst�me a tent� de substituer un lecteur � un r�pertoire stock� sur un lecteur substitu�..
4| ++ => Executable returned with an error code +++++++++++++++++++++++++++++++++++++++++++++++++++++
5| ++ Sending message 0 1 24616 1 T0 I0
6| ++ 9.1.18a ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
7| ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
8| >"\\nadlabnas\Logiciels\RoyalRender\bin\win64\rrKillWait" 1 hserver true
9| ||rrKillWait|| waiting 0.0s for HSERVER.EXE ... thisProcessGroup.
10| ||rrKillWait|| INFO: this process group was started by ... (ID 6236 CMD.EXE)
11| ||rrKillWait|| INFO: parent of our process group starter ... (ID 21712 RRCLIENTCONSOLE.EXE)
12| ||rrKillWait|| no render app with same parent process found. Searching render apps without parent
13| ||rrKillWait|| FOUND process (has parent, IGNORING) HSERVER.EXE ID 6684 owner: SYSTEM| Parent SERVICES.EXE ID1284
14|
Thanks
F
RR, Schoenberger
Hi
I am not aware of a larger change in RR that would prevent Houdini from starting.
Houdini is used a lot with RR and we did not got any issue reports.
Can you start Houdini (via its cached path C:\REZ\houdini\21.0.440-r1\95dd\a\payload\bin\hython.exe) directly and can you start Houdini via Rez without RR?
regards,
Holger Schönberger
Craftsman and Keeper of the Royal Render Flame
Please use the rrKnights Tavern
or our support system for new questions.
Francois Lord
Yes I can run Houdini via rez and hython from the local path. I even tried as the user the service is running under.
I don't think something changed in RR itself, but probably in the way our IT staff installed it in the machines for this year.
I have a hard time troubleshooting this because I don't know what RR is doing exactly before running the process. I copy-pasted the command I see in the log and tried it in a terminal. It runs hython correctly but it fails with this message:
C:\REZ\houdini\21.0.440-r1\95dd\a\payload\bin\hython.exe: can't open file 'C:\\RR_localdata\\renderscripts\\kso_houdini.py': [Errno 2] No such file or directory
F
--
If you reply, the message is send to the user group which is sufficient and desired.
("Reply All" might send the message twice to the last author which is not required.
Replying "in private" prevents other users to see the answers and might not be seen by the single receiver if he has email rules in place.
If you want to talk to us in private, please use support (at) RoyalRender instead)
---
You received this message because you are subscribed to the Google Groups "Royal Render Knights Tavern" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rrKnights+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/rrKnights/004501dc53b1%24b1c22600%2415467200%24%40RoyalRender.de.
RR, Schoenberger
Hi
> can't open file 'C:\\RR_localdata\\renderscripts\\kso_houdini.py
Please change the user rights for the folder C:\\RR_localdata.
Add your user or all users for read/write.
Inherited permissions should already be activated, so it should affect all subfolders and files.
> I have a hard time troubleshooting this because I don't know what RR is doing exactly
After a client got a job, it creates a batch file C:\RR_LocalData\C_...bat
You can execute this batch.
You have to
- Change the job settings: Disable the job option “Local Render out”
- Let the client get the job.
- Uncheck “Enabled” within the rrClient UI to prevent it from getting a new job
- Two ways to start:
a) Start the C:\RR_LocalData\C_...bat from the explorer.
You may add the commandline “pause” to the end of the batch file to prevent it from closing.
b) Open a terminal and execute it.
But in some special cases you may not be able to run the batch twice as some environments are already set.
A clean env is recommended.
You may run this commandline to open a new command prompt:
start cmd /k C_Elliot10_A.bat
One important step in finding the difference is comparing the environment.
Before you do the batch file test
edit the file RR\render_apps\_setenv\all\_global__inhouse.rrEnv
and add the line
[Exec] set
This prints all env vars.
Instead of just running the batch, pipe it into a log file:
start cmd /k “C_Elliot10_A.bat >C:\RR_LocalData\_output.txt”
Use a file comparison app to compare it with the render logfile in RR. (The one that was used to create the batch file)
Get the job {ID} of the job in rrControl
RRs logfile is located in RR/rrJobData/<Project>/{ID-DirstLetter}/{ID}_..../log/
PS:
Houdini 21 is about to be tested for issues in the next few days
Houdini 20.5 was used a lot.
Francois Lord
Ok, so I assigned a single machine to the job, logged-on to it with the same user that is running the service, went in C:\RR_localdata and double clicked on the C_....bat file and it did the same thing as when running from inside RR: it crashed instantly with the message "(0) Couldn't load -/bin/python311.dll."
But this time, when I ran the rez-env command right above the error in a terminal, it ran perfectly fine and the right files were rendered to disk.
I looked at that .bat file and started removing parts and trying it until I got something that worked. I figured out that it part is problematic. When I remove it, it renders fine.
@rem we use this rrEnv file for houdini
and usd_stdA, therefore this switch
echo [If] rrJobRenderapp "%rrJobRenderapp%" [==] Houdini
IF "%rrJobRenderapp%" EQU "Houdini" (
echo [If] rrBaseAppPath "-" [!=]
IF "-" NEQ "" (
echo [If] RR_NO_HFS "%RR_NO_HFS%" [==]
IF "%RR_NO_HFS%" EQU "" (
set "HFS=-"
echo HFS = "-"
set "HH=-\houdini"
echo HH = "-\houdini"
set "HHP=-\houdini\python3.11libs"
echo HHP = "-\houdini\python3.11libs"
)
)
)
F
--
If you reply, the message is send to the user group which is sufficient and desired.
("Reply All" might send the message twice to the last author which is not required.
Replying "in private" prevents other users to see the answers and might not be seen by the single receiver if he has email rules in place.
If you want to talk to us in private, please use support (at) RoyalRender instead)
---
You received this message because you are subscribed to the Google Groups "Royal Render Knights Tavern" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rrKnights+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/rrKnights/000001dc540e%24c358e310%244a0aa930%24%40RoyalRender.de.
RR, Schoenberger
Hi
Yes, the hardcoded version look like an issue.
But it is not be part of our rrEnv files.
By default our submission sends the python version with the job.
If this information is missing (custom submission), the rrEnv assumes the python version from the Houdini version.
And this variable is then used for the python path:
Has the rrEnv file in subfolder \rr_original the right syntax?
What’s the version of the rrEnv file as stated in its first line? And which RR version do you use?
Francois Lord
Our houdini.rrenv looks like yours. The one in rr_original is also the same. Our houdini__inhouse.rrenv only contains 3 env vars.
The _setenv/win folder is empty except for rr_original, which contains houdini.bat, which only has comments.
I did a search in all the "render_apps" folder in the RR network folder for RR_NO_HFS and found 6 files, all of which have the same syntax as your file.
I think the .bat I got was generated by one of those files. How do I know which one was used? Versions of those files vary between v9.1.18 and v9.1.19.
As far as I know, we are not using any custom submission script. I submitted the job from within Houdini with the submission plugin.
In rrControl window title, it says RRv9.1.20 (thisApp 9.1.18a)
F
RR, Schoenberger
Hi
> Our houdini.rrenv looks like yours.
My fault, I mistake the .bat snipplet for the .rrEnv file.
>The _setenv/win folder is empty
Not used any more, you can delete the win folder.
> the .bat I got was generated by one of those files. How do I know which one was used?
rr_original is not used for rendering, it is a backup.
Then the rrClient uses the same rrEnv as the name of the render application. (so in this case houdini.rrEnv)
Note: Each .rrEnv should have an echo line as first entry that is shown in the render log.
Ok, back to the issue.
This is the .rrEnv:
[If] <rrBaseAppPath> [!=]
[If] <RR_NO_HFS> [==]
HFS = <rrBaseAppPath>
HH = <rrBaseAppPath>/houdini
HHP= <rrBaseAppPath>/houdini/python<HPyVerP>libs
[EndIf]
[EndIf]
This is the created batch:
echo [If] rrBaseAppPath "-" [!=]
IF "-" NEQ "" (
echo [If] RR_NO_HFS "%RR_NO_HFS%" [==]
IF "%RR_NO_HFS%" EQU "" (
set "HFS=-"
echo HFS = "-"
set "HH=-\houdini"
echo HH = "-\houdini"
set "HHP=-\houdini\python3.11libs"
echo HHP = "-\houdini\python3.11libs"
)
)
It should have been HHP= C:\REZ\houdini\21.0.440-r1\95dd\a\payload\houdini\houdini\python3.11libs
<rrBaseAppPath> does not work.
I assume the issue is that the the Houdini app is not part of the client config.
And the REZ commandline does not include any path to Houdini that the rrClient can read.
Have to think about how to solve this.
I know that these env vars are rarely used, most scenes work without.
Afaik you need them for TOP/PDG processing, but I do not remember if they are required for other “special stuff” as well.
For now, the best solution is to disable these env vars in the rrEnv file.
Francois Lord
I think I solved the issue on
my side by adding those lines to my houdini rez package.py file:
It recreates the variables Houdini sets in its setup file.
Which solution do you think is better, yours or mine? both?
F
--
If you reply, the message is send to the user group which is sufficient and desired.
("Reply All" might send the message twice to the last author which is not required.
Replying "in private" prevents other users to see the answers and might not be seen by the single receiver if he has email rules in place.
If you want to talk to us in private, please use support (at) RoyalRender instead)
---
You received this message because you are subscribed to the Google Groups "Royal Render Knights Tavern" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rrKnights+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/rrKnights/001301dc5580%240491a160%240db4e420%24%40RoyalRender.de.
RR, Schoenberger
Hi
>Which solution do you think is better, yours or mine? both?
My workaround just removes the env vars if they have not been set.
Which works for most scenes.
=>RR does not set the env var
You have a real fix in case the vars are used.
=>You set the env var
You can use both as they do not interfere.