plctag.dll Still in Use After Shutdown

[Nick Kokocinski]

Hello again,

I'm having an issue with my program when I am shutting down the PLC reading/writing functionality. My program uses the libplctag .NET wrapper compiled into two DLLs, as well as the libplctag compiled DLL.

 In my program I have functionality that, when triggered, updates the code and binaries. When this process is triggered, all active running PLCs are stopped and then the update begins. When I stop the PLCs, I essentially stop polling and call the Dispose method in libplctag .NET (which calls plc_tag_destroy). However, I am running into an issue when I try to update the plctag.dll; it says that it is still in use by my program even though I have stopped reading/writing to any PLC tags and disposed all of them.

Is there something more I have to do to completely stop anything in plctag.dll from executing? It might be worth mentioning that the libplctag .NET DLLs have no issue in updating, it is just plctag.dll.

Thanks in advance,

- Nick

[Kyle]

Hi Nick,

A while back there was another user who had a similar problem when using the C DLL in LabVIEW.   I added a function to the C DLL, plc_tag_shutdown(), to terminate all use of resources within the DLL.     In that case, there was no call to the DLL termination function from the LabVIEW environment.

Try calling that and see if it helps.  It is possible that not everything has terminated when you close all tags.   It can take a little bit of time for all the background threads to terminate, for instance.

Best,

Kyle

[Nick Kokocinski]

Thanks Kyle, I will try that. Do I only have to call it once, or for every PLC tag?

[Nick Kokocinski]

I tried calling shutdown (in libplctag .NET there's a static method called Shutdown() that calls plc_tag_shutdown()), and I'm still getting the same error.

[Kyle]

Try capturing the debug output from the core library.   From that we might be able to tell if there is still something left alive or if it is a Windows issue.

Set debug level to 4 and capture the (large) output.

Best,

Kyle

[Nick Kokocinski]

Where do I set the debug level? And is there anything else I need to configure in order to get the large output?

[Jody Koplo]

So I can't claim to know all the specifics, but I would suspect that .net loads the native dll when it's first used and keeps a handle open to it until the process itself completes. I've seen the same behavior in LabVIEW and other environments. 

I'm guessing you'll need to tell the .net runtime to force the library out of memory - just deallocating items that make use of the library is probably not enough to make it 'let go' of the reference. 

I'll see if I can find some official documentation to validate my hunch...

Jody

--
You received this message because you are subscribed to the Google Groups "libplctag" group.
To unsubscribe from this group and stop receiving emails from it, send an email to libplctag+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/libplctag/31cb2a1d-92b9-49c3-b030-c605b6ddd7a6n%40googlegroups.com.

[Jody Koplo]

I think this is what you want: https://stackoverflow.com/a/66199878/2125462

It seems a bit complicated... And I'm wondering if the native c dll complicates things further, but give that a whirl and see if it does what you need to do.

Jody

[Nick Kokocinski]

Here is the debug output.

[Nick Kokocinski]

Thanks for your response Jody. Is what you're suggesting that, when doing shutdown tasks, I create a new AssemblyLoadContext, load plctag.dll in that content, and then immediately unload?

[Kyle]

Thanks, Nick! Hmm, looks like the library thinks it is shut down correctly:

2022-11-22 10:26:06.214 thread(7) tag(13) INFO plc_tag_destroy:1512 Done.

2022-11-22 10:26:06.268 thread(2) tag(0) DETAIL tag_tickler_func:631 Tag tickler thread timed out waiting for something to do.

2022-11-22 10:26:06.327 thread(8) tag(0) INFO ab_teardown:142 Releasing global AB protocol resources.

2022-11-22 10:26:06.328 thread(8) tag(0) INFO ab_teardown:153 IO thread already stopped.

2022-11-22 10:26:06.328 thread(8) tag(0) INFO ab_teardown:156 Freeing session information.

2022-11-22 10:26:06.328 thread(8) tag(0) DETAIL mutex_destroy:841 destroying mutex 78B2F9E0

2022-11-22 10:26:06.328 thread(8) tag(0) DETAIL mutex_destroy:854 Done.

2022-11-22 10:26:06.328 thread(8) tag(0) INFO ab_teardown:162 Done.

2022-11-22 10:26:06.328 thread(8) tag(0) INFO mb_teardown:2391 Starting.

2022-11-22 10:26:06.328 thread(8) tag(0) DETAIL mb_teardown:2395 Destroying Modbus mutex.

2022-11-22 10:26:06.328 thread(8) tag(0) DETAIL mutex_destroy:841 destroying mutex 78B2F9F4

2022-11-22 10:26:06.328 thread(8) tag(0) DETAIL mutex_destroy:854 Done.

2022-11-22 10:26:06.328 thread(8) tag(0) INFO mb_teardown:2401 Done. 2022-11-22 10:26:06.328 thread(8) tag(0) INFO lib_teardown:171 Tearing down library. 2022-11-22 10:26:06.328 thread(8) tag(0) INFO lib_teardown:176 Signaling tag tickler condition var.

2022-11-22 10:26:06.329 thread(8) tag(0) INFO lib_teardown:181 Tearing down tag tickler thread.

2022-11-22 10:26:06.329 thread(2) tag(0) INFO tag_tickler_func:641 Terminating.

2022-11-22 10:26:06.329 thread(8) tag(0) INFO lib_teardown:188 Tearing down tag tickler condition var.

2022-11-22 10:26:06.329 thread(8) tag(0) DETAIL cond_destroy:1242 Starting.

2022-11-22 10:26:06.329 thread(8) tag(0) DETAIL cond_destroy:1253 Done.

2022-11-22 10:26:06.329 thread(8) tag(0) INFO lib_teardown:194 Tearing down tag lookup mutex.

2022-11-22 10:26:06.329 thread(8) tag(0) DETAIL mutex_destroy:841 destroying mutex 78B2F9A4

2022-11-22 10:26:06.329 thread(8) tag(0) DETAIL mutex_destroy:854 Done.

2022-11-22 10:26:06.329 thread(8) tag(0) INFO lib_teardown:200 Destroying tag hashtable.

2022-11-22 10:26:06.329 thread(8) tag(0) INFO hashtable_destroy:260 Starting

2022-11-22 10:26:06.329 thread(8) tag(0) INFO hashtable_destroy:272 Done

2022-11-22 10:26:06.330 thread(8) tag(0) INFO lib_teardown:207 Done.

2022-11-22 10:26:06.330 thread(8) tag(0) DETAIL mutex_destroy:841 destroying mutex 78B2F998

2022-11-22 10:26:06.330 thread(8) tag(0) DETAIL mutex_destroy:854 Done.

2022-11-22 10:26:06.330 thread(8) tag(0) DETAIL plc_tag_unregister_logger:1303 Starting

2022-11-22 10:26:06.330 thread(8) tag(0) DETAIL plc_tag_unregister_logger:1307 Done.

I see all the resources being cleaned up there.   All the threads indicate that they are being terminated and all the mutexes are freed.

I found this link: https://9to5answer.com/how-to-dynamically-load-and-unload-a-native-dll-file

Maybe in the wrapper for plc_tag_shutdown, the .Net library can do those steps?

Best,

Kyle

[Nick Kokocinski]

I tried doing this myself in the wrapper but it didn't seem to make any difference. I might reach out to the folks on the libplctag .NET github page about this as well, to see if they have a solution for this.

[Jody Koplo]

The GitHub devs are timyhac and me, so you've already got us ;-)

So, to be clear, this isn't something I've done before. It's more that I expect the .net runtime to lazy load dlls and then hold a reference until app shutdown. The SO thread seems to corroborate that.

It looks like you can work around that with some manual context stuff.

Jody

To view this discussion on the web visit https://groups.google.com/d/msgid/libplctag/13bbcdec-caf1-466b-b031-883a260a93cbn%40googlegroups.com.

[Nick Kokocinski]

Alright, thanks! This isn't something I have done before either, so I'm not super familiar with all the details. I modified Shutdown() to call LoadLibrary and then FreeLibrary but this didn't seem to make any difference. Any ideas on how I can modify Shutdown() that will essentially unload the DLL?

[Kyle]

I am still investigating this.   Something about the log trace looks weird to me.   At the top I see plc_tag_destroy() finish.   But that is essentially synchronous.   So it shouldn't be finishing before the teardown of the library even starts.

This hints that there might be a race condition in the shutdown of the library.   Have you tried waiting a second or two between the final plc_tag_destroy() that destroys the last tag and the plc_tag_shutdown() call?

More things to do this weekend!

Best,

Kyle

[Nick Kokocinski]

Hey Kyle, thanks for the update. I tried sleeping for a few seconds before calling Shutdown but that didn't seem to make a difference; I'm still getting the same error.

[Kyle]

I am working on some code changes that make plc_tag_shutdown() go a lot deeper to kill off every allocated resource.   I was testing it when my power went out for several hours :-/  The lights are back on, so I am back testing.   Looks good so far.

This is a change in behavior (though probably one that people thought was happening) for plc_tag_shutdown().   Before, I said that it would leak resources if you did not close all tags before you called it.   The problem is that some resources can be in use by internal threads and won't clean up until those threads finally stop.   And that happened after plc_tag_shutdown() returned from being called.  Now it will forcibly kill off all open tag handles, wait for all threads to terminate and release all core library resources before it returns.

It just passed the valgrind test for memory leaks under multithreading.  Now, how can we see if this helps?

Another thing I found out is that when you load and use a DLL in Windows, it looks like there a sort of reference count going on.   You cannot unload the DLL until the count goes to zero.  

I have a version of the shutdown changes in the prerelease branch.  It will be released as version 2.5.5.

Best,

Kyle

[Nick Kokocinski]

Hey Kyle,

I've tried doing some testing with the latest version, but I am still getting the same error about the DLL being in use as before.

[Kyle]

As I mentioned above, I think there might be some sort of reference count that holds the library in memory.   At this point, it appears that all the resources that the library is using are released, so there isn't any thread or anything that is left over.   But, the libplctag.Net native layer may be still holding on to the C DLL. 

Do we know (this is s question to Jody and Tim) if the number of dllOpen() calls is the same as closes?

Best,

Kyle

On Monday, December 5, 2022 at 9:19:46 AM UTC-8 nick.ko...@runb3.com wrote:

Hey Kyle,

[Nick Kokocinski]

Hello,

Just wondering if anybody has an update on this.

Thanks!

[tim...@gmail.com]

Hi Nick - Kyle raised an issue on your behalf on the .NET wrapper github repository:

https://github.com/libplctag/libplctag.NET/issues/317

Tim

[Nick Kokocinski]

Ah, thank you. Will keep an eye on that thread.