'GstBus' Thread Leaking

[Jacob]

Hi,

When starting a pipeline multiple times from one single application instance, it looks like there are Threads leaking. These threads are called "GstBus" according to the program "VisualVM". (See screenshots)

Strange thing is, when I remove the 'Thread.sleep(1000);' from the example code below, the application seems stable, and no more than 3 "GstBus" threads are created.

I'm using JNA 5.2.0 and gst1-java-core 0.9.4.

Am I doing something wrong? Or is this a bug? 

Greetings. 

Example Code: 

import org.freedesktop.gstreamer.Bus;
import org.freedesktop.gstreamer.Gst;
import org.freedesktop.gstreamer.Pipeline;

import static java.lang.Thread.sleep;

public class TestMain {
 public static void main(String[] args) throws InterruptedException {
 Gst.init();

 for(int loop = 0; loop < 500; ++loop) {

 String pipelinestring = "videotestsrc num-buffers=100 ! " + "x264enc speed-preset=0 ! h264parse ! fakesink";

 Pipeline pipeline = (Pipeline) Gst.parseLaunch(pipelinestring);
 pipeline.getBus().connect((Bus.ERROR) (gstObject, i, s) -> System.out.println("ERROR "+i+" "+s+" "+gstObject));
 pipeline.getBus().connect((Bus.WARNING) (gstObject, i, s) -> System.out.println("WARN "+i+" "+s+" "+gstObject));
 pipeline.getBus().connect((Bus.EOS) obj -> {
 System.out.println("EOS " + obj);

 ((Pipeline) obj).stop();
 ((Pipeline)obj).getBus().dispose();
 obj.dispose();
 Gst.quit();
 obj = null;

 System.gc();
 });

 pipeline.play();

 Gst.main();

 // Disable this to have fixed GstBus Threads
 Thread.sleep(1000);
 }

 // Block to see output
 while(true){
 sleep(10);
 }
 }
}

With delay:

Without delay:

[Neil C Smith]

Hi,

On Fri, 8 Feb 2019 at 18:30, Jacob <Jacob....@gmail.com> wrote:
> When starting a pipeline multiple times from one single application instance, it looks like there are Threads leaking. These threads are called "GstBus" according to the program "VisualVM" ... Am I doing something wrong? Or is this a bug?

Well I'm not entirely sure what you're trying to do! But these aren't
threads. They're Thread proxies created by the VM when native threads
from GStreamer attach. It might be that they're taking a bit of time
to clean up. Once pipelines are destroyed does forcing GC from
VisualVM get them to clean up?

Best wishes,

Neil

--
Neil C Smith
Artist & Technologist
www.neilcsmith.net

PraxisLIVE - hybrid visual live programming
for creatives, for programmers, for students, for tinkerers
www.praxislive.org

[Jacob]

Well, I'm trying to spawn a GStreamer pipeline on demand. I want these to run non-blocking from the rest of the application. 

This is working fine, except for these Thread proxies staying alive, which I think will become a problem in the long run. 

They aren't cleaned up by forcing GC. Are the pipelines destroyed right in the example code? 

Thanks for the quick reply! 

Op vrijdag 8 februari 2019 19:49:35 UTC+1 schreef Neil C Smith:

[Neil C Smith]

On Fri, 8 Feb 2019, 19:41 Jacob <Jacob....@gmail.com wrote:

This is working fine, except for these Thread proxies staying alive, which I think will become a problem in the long run.

I'm not sure you'll see this in practice in an application, and this sort of thing has been tested for. But offhand I'm not sure why you're seeing this with the code you have here either. It might be an issue with JNA's callback initializer code for VM detachment. Possibly also test with JNA 4.5.2.

They aren't cleaned up by forcing GC. Are the pipelines destroyed right in the example code?

You don't need to manually dispose the pipelines at all. Maybe remove the disposal and see if the bug still occurs.

Best wishes, 

Neil

[Jacob]

Removed the dispose functions, and switched to JNA 4.5.2. Does not seem to make a difference. 

Is this kind of pipeline creation strange behaviour? Seems to me you should be able to create and more than one pipeline.. 

What would other options be? Creating a separate application in which the pipeline is played and then start that whole application multiple times?

Best regards,

Jacob

Op vrijdag 8 februari 2019 23:00:04 UTC+1 schreef Neil C Smith:

[Neil C Smith]

On Fri, 8 Feb 2019, 22:51 Jacob <jacob....@gmail.com wrote:

Is this kind of pipeline creation strange behaviour? Seems to me you should be able to create and more than one pipeline.. 

Yes, and I often do, as do many other users of the bindings. So what exactly is going on here I'm not sure, but will try and have a look next week. What OS, Java etc. versions are you using? Do those GstBus threads eventually clear out if you wait a while? 

Best wishes, 

Neil

[Jacob]

Versions:

• macOS 10.14 (18A391)
• java 11.0.1 2018-10-16 LTS
• GStreamer 1.14.4
• gst1-java-core 0.9.4
• jna 5.2.0

I'll experiment some more with the timeout between pipeline creations. Until now the GstBus thread didn't seem to disappear over time. (~ 1 hour)

Op zaterdag 9 februari 2019 00:10:48 UTC+1 schreef Neil C Smith:

[Neil C Smith]

On Fri, 8 Feb 2019 at 23:44, Jacob <jacob....@gmail.com> wrote:
>
> Versions:
>
> macOS 10.14 (18A391)
> java 11.0.1 2018-10-16 LTS
> GStreamer 1.14.4
> gst1-java-core 0.9.4
> jna 5.2.0

OK, I've tried your example on Linux (Ubuntu 18.04) with OpenJDK 8 and
cannot replicate the problem. I think it might be macOS specific, and
might be an issue with how pthread destructors are being handled and
not triggering detachment in JNA. There were some historic issues
here. Although interestingly I also don't see multiple GstBus threads
which suggests some reuse of native threads is going on - need to look
further into this.

I did notice an issue with the Bus not being manually disposed causing
a memory leak though, and have opened an issue for it. May need to
consider disposing the Bus when the Pipeline is disposed, although
with the memory handling changes coming soon this might be harder to
manage.

[Jacob]

Interesting! I'll do some tests with Linux OS's. Strange thing is that when you remove the delay from the example, threads also seem to be reused in macOS.

So you're saying manually disposing the Bus is needed right now? 

Best regards.

Op zaterdag 9 februari 2019 15:20:44 UTC+1 schreef Neil C Smith:

[Neil C Smith]

Hi,

On Sat, 9 Feb 2019 at 19:21, Jacob <jacob....@gmail.com> wrote:
> Interesting! I'll do some tests with Linux OS's. Strange thing is that when you remove the delay from the example, threads also seem to be reused in macOS.

Adding the following code helps show when the threads are reused and not -

pipeline.getBus().setSyncHandler((message) -> {
Thread t = Thread.currentThread();
if ("GstBus".equals(t.getName())) {
System.out.println("" + t.getName() + " : " + t.getId());
}
return BusSyncReply.PASS;
});

This needs to be done in a sync handler as handlers added via
connect() are called on another thread.

With the Thread.sleep() I get multiple GstBus threads, but they're
cleaned up quite quickly. What about if you monitor the number of
instances of java.lang.Thread in the Sampler / Memory tab. Do you get
a continually rising number of Thread instances? If so, it seems that
the CallbackThreadInitializer code in JNA is not working correctly on
mac. I'll consider a System Property switch, but worth reporting
upstream too.

> So you're saying manually disposing the Bus is needed right now?

Yes, seems to be until v1.0 is finished.

[Jacob]

Hi,

I see threads being reused, both with delay and without delay. Difference is, without delay only 2 threads are created, with delay they seem to increase.

I see almost the same behaviour on Ubuntu 18.04, but there the threads get cleaned up nicely. 

Without delay:

GstBus : 19
GstBus : 19
GstBus : 19
GstBus : 19
GstBus : 19
GstBus : 19
GstBus : 16
GstBus : 16
GstBus : 16
GstBus : 16
GstBus : 16
GstBus : 16
GstBus : 16
GstBus : 16
GstBus : 16
GstBus : 16
GstBus : 16
GstBus : 16
GstBus : 16
GstBus : 16
GstBus : 16
GstBus : 19
GstBus : 19
GstBus : 19
GstBus : 19
GstBus : 19
GstBus : 19
GstBus : 16
GstBus : 16
GstBus : 16
GstBus : 16
GstBus : 16
GstBus : 16
GstBus : 16

With delay:

GstBus : 219
GstBus : 219
GstBus : 219
GstBus : 219
GstBus : 219
GstBus : 219
GstBus : 219
GstBus : 219
GstBus : 219
GstBus : 221
GstBus : 221
GstBus : 221
GstBus : 221
GstBus : 221
GstBus : 221
GstBus : 219
GstBus : 219
GstBus : 219
GstBus : 219
GstBus : 221
GstBus : 221
GstBus : 221
GstBus : 221
GstBus : 221
GstBus : 221
GstBus : 221
GstBus : 221
GstBus : 221
GstBus : 223
GstBus : 223
GstBus : 223
GstBus : 223
GstBus : 223
GstBus : 223
GstBus : 221
GstBus : 221
GstBus : 221
GstBus : 221

I do not see any increase in the 'instances of java.lang.Thread' instances. 

MacOS:

Ubuntu:

Op maandag 11 februari 2019 18:51:54 UTC+1 schreef Neil C Smith:

[Neil C Smith]

On Mon, 25 Feb 2019 at 11:17, Jacob <jacob....@gmail.com> wrote:
> I see threads being reused, both with delay and without delay. Difference is, without delay only 2 threads are created, with delay they seem to increase.

Thanks! There are multiple calls through the same Bus there, so
you're not seeing the same thread being reused between buses if the ID
increases.

It looks like we have an issue with callback detachment on macOS. I
think you should ask upstream on this issue, but I'll also look to add
a system property to control use of CallbackThreadInitializers.

[vft...@gmail.com]

Hi everyone, 

I'm using GStreamer for E2E tests and I also realized the `GstBus` leak.

In my case, I'm not reusing the same pipeline, and I always call pipeline stop and dispose but the threads are still there 

I was reading about CallbackThreadInitializer and I wonder if we could somehow call Native.detach() to detach the thread.

https://java-native-access.github.io/jna/4.2.1/com/sun/jna/CallbackThreadInitializer.html

https://java-native-access.github.io/jna/4.2.1/com/sun/jna/Native.html#detach-boolean-

https://github.com/gstreamer-java/gst1-java-core/blob/55ed120b595f05831eeb798113bb6bc1f8e96c5f/src/org/freedesktop/gstreamer/Bus.java#L721

[Neil C Smith]

Hi,

On Fri, 8 Mar 2019, 20:02 , <vft...@gmail.com> wrote:

I'm using GStreamer for E2E tests and I also realized the `GstBus` leak.

In my case, I'm not reusing the same pipeline, and I always call pipeline stop and dispose but the threads are still there

Using dispose explicitly is not intended to fix this. What OS is this on? MacOS? I think we need to see what the common denominator is and then report upstream with JNA.

Incidentally, explicit dispose() has an unrelated issue in the current beta that will be fixed next week. 

I was reading about CallbackThreadInitializer and I wonder if we could somehow call Native.detach() to detach the thread..

This might be awkward as you have to call it on that thread and not a lot of user code actually runs there. And we'd need to consider if we get useful messages on that thread after shutting down the pipeline in which to do this. 

Please open an issue on GitHub about this, and maybe also about a system property for controlling use of the callback thread initializer, and we can manage this there. 

I hope that isn't the version of JNA you're actually using! 

Best wishes, 

Neil

[vft...@gmail.com]

Thanks for the quick reply!

I'm on macOS High Sierra and currently using JNA 5.2.0 and gstreamer 1.0.0-beta-1. But I'm planning to test on Ubuntu on Monday.

I tried a bunch of stuff today and ended up here, but my main issue is if I have too many tests using GStreamer (I'm using the new awesome WebRTC implementation) I end up being flooded with errors like

(gst1-java-core:37977): GLib-WARNING **: 16:33:22.568: poll(2) failed due to: Invalid argument.

Doing some profiling I saw a bunch of objects that weren't being clean and I mitigated the issue by calling `System.gc()` directly but my number of threads constantly increases during my executions and it's mostly GstBus.

Other things I tried was to call pipeline.close(), pipeline.dispose(), Gst.quit() and Gst.deinit() but without much success.

If you think that the Native.detach() makes sense I will gladly open an issue on Github, and also try to help any way I can 

Regards

[Neil C Smith]

On Fri, 8 Mar 2019 at 21:39, <vft...@gmail.com> wrote:
> I'm on macOS High Sierra and currently using JNA 5.2.0 and gstreamer 1.0.0-beta-1. But I'm planning to test on Ubuntu on Monday.

OK, thanks, so macOS does seem to be the common denominator here.
Which Java version are you using btw?

> I tried a bunch of stuff today and ended up here, but my main issue is if I have too many tests using GStreamer (I'm using the new awesome WebRTC implementation) I end up being flooded with errors like
>
> (gst1-java-core:37977): GLib-WARNING **: 16:33:22.568: poll(2) failed due to: Invalid argument.

That's an error in the native side. Do you know if there's a specific
method you're calling that triggers it? How many concurrent tests are
you trying to run? Have you searched online for that message and
explored the various things referred to?

> Other things I tried was to call pipeline.close(), pipeline.dispose(), Gst.quit() and Gst.deinit() but without much success.

There's currently an issue with dispose() in the beta that means a
strong reference is kept in a small number of cases -
https://github.com/gstreamer-java/gst1-java-core/issues/146 It will
work correctly without explicit dispose(). I need to work out the
exact case so we can also add a test for this - it currently passes
the GC tests.

DO NOT use Gst.deinit() -
https://github.com/gstreamer-java/gst1-java-core/blob/master/src/org/freedesktop/gstreamer/Gst.java#L530

> If you think that the Native.detach() makes sense I will gladly open an issue on Github, and also try to help any way I can

The initial thing will be a system property to switch off the
CallbackThreadInitializers. You will then see *a lot* of threads
being created, but they'll be disposed correctly almost immediately.
Then we can look if it's even feasible to use Native.detach() and/or
discuss upstream. There seems to be a bug in JNA's use of pthread
destructors on macOS.

[vft...@gmail.com]

I'm using Java openjdk version "11" 2018-09-25

For what I saw, the issue is triggered on pipeline.play().

Without the System.gc() I can run about 10 tests (And interestingly enough if I don't call pipeline.dispose(), the limit is also around 10). With GC and dispose, I can run up to 50 before the issue starts.

I did google about the error and it seems related to me exceeding the number of file descriptors. And I just realized my mistake considering threads = processes :p

I also created an issue as we discussed: https://github.com/gstreamer-java/gst1-java-core/issues/147

Now I will look into reducing the numbers of sockets.

I realized I have UDP and TCP sockets being opened and not cleared as well.