AE2018 Errors

[Magnetic Dreams]

Hi,

This is an error that we have seen over the years, but is typically quite rare and gets ignored. However, recently we are seeing this even on the simplest of scenes with no effects. I understand this error is not RR specific, since it's an aerender error, but I feel that RR maybe triggering or causing aerender to somehow produce it.

I have tried to reproduce this error by command line on the problematic systems, but I am unable to reproduce and it renders without any problems. There appears no logic to which systems are affected, but typically our workstations (even with all Adobe products closed) give the error. Some systems render without the error. If we enable "Ignore Crashes", then we can ignore it, but the crashes still slow us down. The error suggest we don't use "-reuse", but this is already not used by default, and should create a new instance of aerender.


R 53| aerender ERROR An existing connection was forcibly closed by the remote host.

R 54| : Unable to receive at line 314

R 55| aerender ERROR: After Effects can not render for aerender. Another instance of aerender, or another script, may be running; or, AE may be waiting for response from a modal dialog, or for a render to complete. Try running aerender without the -reuse flag to invoke a separate instance of After Effects

After Effects 2018 (15.0.1)

RR 8.2.07b

Any ideas? does anyone else receive these errors?

Thanks.

[RR, Schoenberger]

Hi

 

The main difference between running it in a command prompt and running it via RR ist hat RR runs as background service.

We have noticed that there are even some projects that do not render at all if the client is running as service.

If you start the rrClient in application mode, then Afx renders fine.

 

We get your error message if we try to start multiple AFX at the same time on our machines.

The message itself is not right at all. Just assumptions what could be wrong.

We already tried to contact Adobe via their support channels, but there is no interest. Some resellers we know tried to contact Adobe as well without any result.

 

 

regards,
Holger Schönberger

 

Please use the rrKnights Tavern
or our support system for new questions.

--
If you reply, the message is send to the user group which is sufficient.
("Reply All" just sends the message twice to the last author which is not required)
---
You received this message because you are subscribed to the Google Groups "Royal Render Knights Tavern" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rrKnights+...@googlegroups.com.
Visit this group at https://groups.google.com/group/rrKnights.
For more options, visit https://groups.google.com/d/optout.

[Magnetic Dreams]

The machines that are producing the errors are not running as a background service, RR is already running in application mode. The errors became more frequent at the time we went to CC2018.

It's almost like RR is trying to start the aerender.exe process again, even though it is already running and triggering the error. Almost like RR does not realize the process is already active. If the bug was with aerender itself, I feel that it could be reproduced locally. It could just be aerender being buggy since 2018 made it worse, or a combination of AE and RR, I really have no idea!

[RR support, Schoenberger]

Just to confirm: The Adobe cloud services are all running?


I can imagine 3 cases in which multiple AFX are running:
1) The client has multiple job threads as set in rrConfig.
2) You have enabled the Multiple Instance option, in this case you get
2 render logs for the same job at the same time.
3) Someone is starting the After Effects UI to work.

Or it is an AFX related process/background service that needs some
time after the old aerender instance was closed.

Note: The first part is the real error message (and the 2 lines before that):

"aerender ERROR: After Effects can not render for aerender."

The second part is just an assumption ("may be running"). From our
experience it is not related to the error itself. For example we never
used the -reuse flag.

"Another instance of aerender, or another script, may be running; or,
AE may be waiting for response from a modal dialog, or for a render to
complete. Try running aerender without the -reuse flag to invoke a

separate instance of After Effects."


I will log a dev ticket to investigate AFX in deep (probably next month).
This case is related to some TCP communication between processes.
(I assume afx does not directly "call home", it connects to the
license service). Perhaps a TCP sniffer like Wireshark tells us which
process it tries to connect to. And if we know the process, it might
be possible to restart it before Afx is launched.

[Magnetic Dreams]

Correct, all cloud services for adobe and the cloud app are all running and signed-in on these systems.

1) No, all the machines have job threads set to 1.
2) No, on the job settings Multiple Instance is set to 1.
3) No, we receive the error on machines that are not being used, all Adobe products are closed during rendering.

It is possible there is a dependency of aerender.exe that is not closing correctly (but not aerender.exe itself), after a previous render?
I'll run some more tests on this end to see if i can figure out any pattern. Please let me know if you figure anything out with your tests!

Thanks.

[Magnetic Dreams]

I'm uploading a debugInfo zip file to you now. This was a very simple test I just did, converting a .tiff sequences into a .png sequence. Although the job rendered, many of the machines gave errors and crashes.

[Magnetic Dreams]

I'm also noticing in the machine logs that it outputs this error many times. Not sure if this is related or triggers the crash, or maybe another problem altogether.

Mar 22. 15:10.21| WRN  Unable to load function 'imgLoadHeader' from library rrImagePng64.dll: Error 127: The specified procedure could not be found..
Mar 22. 15:10.21|     {54N } Image load failed for file(2) C:\RR_localdata\renderout\A\TEMP\james\test\test_00000.png Unable to load ''!

[RR, Schoenberger]

I will re-add the plugin to the next update.

Seems it got missed at some point.

 

regards,
Holger Schönberger

 

Please use the rrKnights Tavern

or our support system for new questions.

From: rrkn...@googlegroups.com [mailto:rrkn...@googlegroups.com] On Behalf Of Magnetic Dreams
Sent: Thursday, March 22, 2018 9:29 PM
To: Royal Render Knights Tavern <rrKn...@googlegroups.com>

Subject: Re: [rrKnights] AE2018 Errors

[Magnetic Dreams]

Good news!

I have figured out the source of the random AE render crashes! I managed to isolate the issue to a plugin we have in our workgroup. Without the plugin the renders are rock solid, with no errors at all. With the plugin, random crashes consistently occur. It causes the aerender.exe/AfterFX.com processes to become unstable.

The plugin is VideoCopilot's FXConsole.aex. We are using the latest version (1.0.2), which was built specifically for AE2018 in October.

I will see if i can contact Andrew Kramer, and maybe he has a fix for this.

[RR support, Schoenberger]

Does the plugin use a license server?
As the error message was TCP related.
But with AFX I could imagine that this message is wrong as well.

[Magnetic Dreams]

No, FXConsole is a free plugin from Video Copilot, there is no licensing.
They are planning on sending us a debug build, to output crash log info.

[Magnetic Dreams]

Just to let you know, we received a beta build of FXConsole (v1.0.3b3) today from VideoCopilot, and all the AE render errors are now gone!

[RR, Schoenberger]

Nice.

So it was some kind of bug in the plugin.

Does he know what might have caused it?

 

And it confirms that the AFX error messages are just wrong, even the TCP error message.

 

regards,
Holger Schönberger

 

Please use the rrKnights Tavern

or our support system for new questions.

From: rrkn...@googlegroups.com [mailto:rrkn...@googlegroups.com] On Behalf Of Magnetic Dreams
Sent: Tuesday, March 27, 2018 10:22 PM
To: Royal Render Knights Tavern <rrKn...@googlegroups.com>
Subject: Re: [rrKnights] AE2018 Errors

 

[Magnetic Dreams]

No idea, my response was not from Andrew, so I did not get any details on exactly what was fixed.

[Magnetic Dreams]

I asked for more information on this, and the response was...

"I can tell you how I fixed it, but I suspect it will not help Holger in any way. The solution I found was to disable the plugin when no After Effects UI is present. The thing is, this plugin is a bit different than usual. In order for us to be able to open the console with a key combo, we had to install what Adobe calls an IdleHook, this is kind of an infinite loop function where we look for keyboard inputs. I just disabled this "hook" since no UI is present when you render using RoyalRender and aerender.exe from the console. The plugin is still initialized, but without an UI and without the loop.

As for any of our plugins, we already have a system that catches crashes and saves dump files for us to check later. However, the crash you reported was something completely different that wasn't caught by our handler, so there was nothing I could add that could help us see exactly where the problem was. As I mentioned before, the crash occurs within AE itself, even though it seems to be related to the presence of the FX Console.

For me to really find the culprit, I had to be able to get the crash on my system, because debugging via email in a user pc is not a viable option"

[RR, Schoenberger]

Thanks, it helps me a bit.

In case an other plugin has the same issues.

[Magnetic Dreams]

On the latest version, we are still seeing PNG DLL errors on the rrClient's, for example...

WRN  Unable to load function 'imgLoadHeader' from library rrImagePng64.dll: Error 127: The specified procedure could not be found..

Image load failed for file(2) C:\RR_localdata\renderout\A\AppUIs\Output\Comp\001\001_AppUI216.png Unable to load "!

[RR, Schoenberger]

I have just run a test with .png files on Windows without issue.

And the 8.2.08 update should have updated the plugin in case it was an old version.

What is the file date of the file  RR\plugins64\image\rrImagePng64.dll?

 

Can you use the check button in rrControl to view the files? It uses the same plugin.

[Magnetic Dreams]

The date on the file is 30-March-2018 @ 3:39pm
The control works fine, thumbnails are good, and the rrViewer has no problems with the png's.

[RR support, Schoenberger]

> The date on the file is 30-March-2018 @ 3:39pm

That is the file of the last update.

Perhaps a local cache cannot be replaced due to an user permission error.
(or the client was not restarted, but the update should have restarted it).

Is the client running as service, then please check the plugin in
c:\Program Files\rrService\lc64\image\
If it was started in manually, please check the folder
%TEMP%\rrbin64\image

If it is not the same file, please delete it and restart the client.

[Magnetic Dreams]

Client were restarted, and shows 8.2.08 on the title.
Clients are started manually, inside %TEMP%\rrbin64 there are only 3 files, and no \image folder.

rrResetexitcode.exe
rrShared.dll
rrShared.dll_old

However, inside %TEMP%\rrbin, there is an image folder, which contains...

rrImageArchive64.dll
rrImageExr64.dll
rrImagePng64.dll

Maybe this is the problem? the 64-bit plugins are copied to the rrbin folder, instead of rrbin64?

[Magnetic Dreams]

the date on the local dll's in the \image folder are also 3/30/2018.

[RR support, Schoenberger]

You are starting the rrClient, not rrClientconsole, right?
The rrClient.exe is 64bit only.
So if the client reports that it cannot use rrImagePng64.dll,
then tries to load the right version.

But you are right, the files should be in the rrbin64 folder, this
will be fixed.

But there should be 16 image plugins in that folder, not 3.
Do you start the client with the win__rrClient.bat (which uses
startlocal) or do you start them directly from RR/bin/win64?

[Magnetic Dreams]

Correct, it's the rrClient that we always start by launching win_rrClient.bat
I checked several other clients, and only seeing the same 3 files locally, within the "%TEMP%\rrbin\image" folder. If i check the rrClient.exe process in the task manager, the Command line path points to: "%TEMP%\rrbin\rrClient.exe -rrRoot \\rrserv\rr". If the client is 64-bit only,  i would expect to see it in the rrbin64 folder.

Full contents of the rrbin and rrbin64 folders on the clients is as follows:

%TEMP%\rrbin\image\rrImageArchive64.dll
%TEMP%\rrbin\image\rrImageExr64.dll
%TEMP%\rrbin\image\rrImagePng64.dll
%TEMP%\rrbin\Half.dll
%TEMP%\rrbin\Iex-2_2.dll
%TEMP%\rrbin\IlmImf-2_2.dll
%TEMP%\rrbin\IlmThread-2_2.dll
%TEMP%\rrbin\Imath-2_2.dll
%TEMP%\rrbin\libeay32.dll
%TEMP%\rrbin\msvcp120.dll
%TEMP%\rrbin\msvcr120.dll
%TEMP%\rrbin\QtCore4.dll
%TEMP%\rrbin\QtGui4.dll
%TEMP%\rrbin\QtNetwork4.dll
%TEMP%\rrbin\rrClient.exe
%TEMP%\rrbin\rrClient.exe_old
%TEMP%\rrbin\rrJpeg.dll
%TEMP%\rrbin\rrShared.dll
%TEMP%\rrbin\rrShared.dll_old
%TEMP%\rrbin\ssleay32.dll

%TEMP%\rrbin64\rrResetexitcode.exe
%TEMP%\rrbin64\rrShared.dll
%TEMP%\rrbin64\rrShared.dll_old

[RR, Schoenberger]

But there are more files in the folder RR/plugins64/image?

Please shutdown and start the client with win_rrClient.bat again.

Did it copy more than 3 files now?

 

Do you use any script to clear the temp folder regulary?

The rrbin/rrbin64 issue is  a bug in the 64bit version of the startlocal application.

It does not matter from which folder the rrClient is started, but fixed anyway for the next version.

The bug is not in the rrClient, so additional tools used by the client are copied into the folder /rrbin64

 

regards,
Holger Schönberger

 

Please use the rrKnights Tavern

or our support system for new questions.

[Magnetic Dreams]

Once the clients were restarted, then the rrbin\image folder became populated with all 16 dll's.
Not sure why they were not there before, since the update should have told the server to restart the clients.

[Magnetic Dreams]

Regarding the random errors we had earlier...
Once the FX Console plugin was fixed by VideoCopilot, the errors went away, at least on basic test renders. However, we soon discovered that more complex comps continued to have the same random errors, even with the new FX Console.

After many test renders and many hours of diagnosis, I've been able to isolate the error to usage of the "Fast Blur (Legacy)" effect in After Effects. This obsolete plugin causes serious instabilities with the 2018 aerender.exe. Hope this info helps someone else out!

[Magnetic Dreams]

Just to add to this, we have since discovered the using the "Fast Box Blur" effect ALSO causes AE2018 to generate random crashes in a network rendering environment. So, avoid using any type of Fast Blur effect in AE2018 to eliminate the crashes.

[Magnetic Dreams]

Yesterday we discovered two more effects that also cause the crashes:

Simple Choker
Directional Blur

If we avoid these effects altogether then everything is fine. When using any of these effects it will cause a crash depending on the amount of VRAM (GPU RAM) free in the system. I am able to cause a command line render to work or fail, depending on how much VRAM is available.

However! If the same job is sent to RR, even with the VRAM level low on the assigned system, we are still experiencing crashes using these effects. I'm convinced that the Adobe issue is somehow triggering an RR issue.

I have just uploaded you the debug file from the job. This is a simple comp using all four of the mention effects, and assigned to one system (Beaky). This system has a 4GB GPU, which was low usage at time of render.

I've also posted the command line issue to Adobe...
https://forums.adobe.com/thread/2486191

[RR, Schoenberger]

Hi

 

Thanks for the information.

 

I will check the debug file in case I find something new.