A lot of short-lived thread while using gstreamer-java

[saaleb]

Hi,

During load test of GStreamer-java-based application I found a lot of short-lived java thread (about 30) that occurs on every GST job start. In case of one-two simultaneous processing it have no effect, but in case of ~100 jobs it have a huge overhead on threads creation(context switching and so on). 

Typical call tree of this thread:

100,0% - 3 040 µs - 1 inv. com.sun.jna.CallbackReference$DefaultCallbackProxy.callback

78,3% - 2 381 µs - 2 inv. org.gstreamer.lowlevel.GTypeMapper$3.fromNative

76,6% - 2 329 µs - 2 inv. org.gstreamer.lowlevel.NativeObject.objectFor

0,4% - 11 µs - 2 inv. com.sun.jna.FromNativeContext.getTargetType

21,2% - 643 µs - 1 inv. org.gstreamer.Bus$2.callback

11,0% - 334 µs - 1 inv. org.gstreamer.$Proxy4.gst_mini_object_unref

7,1% - 216 µs - 1 inv. java.util.concurrent.Executor.execute

0,4% - 13 µs - 1 inv. org.gstreamer.Gst.getExecutor

0,2% - 7 µs - 1 inv. org.gstreamer.Bus.access$2

0,2% - 5 µs - 1 inv. org.gstreamer.BusSyncReply.intValue

0,2% - 5 µs - 1 inv. org.gstreamer.Bus$2$1.<init>

0,2% - 5 µs - 1 inv. org.gstreamer.Bus.access$0

0,1% - 4 µs - 1 inv. org.gstreamer.Bus$1.syncMessage

Have anybody collides with such situation and how can I improve a performance of GStreamer-java part?

Thanks!

[Neil C Smith]

Hi,

At a guess this is related to JNA callbacks - see eg. here -
https://groups.google.com/d/topic/gstreamer-java/DXojRl6R2CM/discussion

If you're using JNA 3.4.0+ try adding Native.detach(false); in one of
your listeners and see if this fixes the issue. To fix properly this
really requires the thread to be detached on exit, which is automated
in JNA 3.5.0. However, I have a feeling there are Structure mapping
issues between GStreamer-Java and JNA 3.5.0 at the moment?

Best wishes,

Neil

On 5 November 2012 14:44, saaleb <ybalt...@gmail.com> wrote:
>
> Hi,
>
> During load test of GStreamer-java-based application I found a lot of short-lived java thread (about 30) that occurs on every GST job start. In case of one-two simultaneous processing it have no effect, but in case of ~100 jobs it have a huge overhead on threads creation(context switching and so on).
>
> Typical call tree of this thread:
>

> 100,0% - 3 040 µs - 1 inv. com.sun.jna.CallbackReference$DefaultCallbackProxy.callback

> 78,3% - 2 381 µs - 2 inv. org.gstreamer.lowlevel.GTypeMapper$3.fromNative

> 76,6% - 2 329 µs - 2 inv. org.gstreamer.lowlevel.NativeObject.objectFor

> 0,4% - 11 µs - 2 inv. com.sun.jna.FromNativeContext.getTargetType

> 21,2% - 643 µs - 1 inv. org.gstreamer.Bus$2.callback

> 11,0% - 334 µs - 1 inv. org.gstreamer.$Proxy4.gst_mini_object_unref

> 7,1% - 216 µs - 1 inv. java.util.concurrent.Executor.execute

> 0,4% - 13 µs - 1 inv. org.gstreamer.Gst.getExecutor

> 0,2% - 7 µs - 1 inv. org.gstreamer.Bus.access$2

> 0,2% - 5 µs - 1 inv. org.gstreamer.BusSyncReply.intValue

> 0,2% - 5 µs - 1 inv. org.gstreamer.Bus$2$1.<init>

> 0,2% - 5 µs - 1 inv. org.gstreamer.Bus.access$0

> 0,1% - 4 µs - 1 inv. org.gstreamer.Bus$1.syncMessage
>
> Have anybody collides with such situation and how can I improve a performance of GStreamer-java part?
>
> Thanks!
>

> --
> You received this message because you are subscribed to the Google Groups "gstreamer-java" group.
> To view this discussion on the web visit https://groups.google.com/d/msg/gstreamer-java/-/ZwOPOK3bMiYJ.
> To post to this group, send email to gstream...@googlegroups.com.
> To unsubscribe from this group, send email to gstreamer-jav...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/gstreamer-java?hl=en.




--
Neil C Smith
Artist : Technologist : Adviser
http://neilcsmith.net

Praxis LIVE - open-source, graphical environment for rapid development
of intermedia performance tools, projections and interactive spaces -
http://code.google.com/p/praxis

OpenEye - specialist web solutions for the cultural, education,
charitable and local government sectors - http://openeye.info

[saaleb]

HI,

Thanks for quick response. I tried to update JNA to latest (3.5.1) and really found a error with Structure mapping between JNA and GStreamer-java

So the way to fix my issue, as I realized, is to wait for Structure mapping issue resolve or trying to patch GStreamer-java for thread detaching.

Not sure it may be done in my java code, unfortunatelly.

понедельник, 5 ноября 2012 г., 17:26:20 UTC+2 пользователь neilcsmith.net написал:

[Neil C Smith]

On 6 November 2012 10:57, saaleb <ybalt...@gmail.com> wrote:
> Thanks for quick response. I tried to update JNA to latest (3.5.1) and
> really found a error with Structure mapping between JNA and GStreamer-java
> So the way to fix my issue, as I realized, is to wait for Structure mapping
> issue resolve or trying to patch GStreamer-java for thread detaching.
> Not sure it may be done in my java code, unfortunatelly.
>

JNA 3.5.1 will not by itself fix this issue - it automates thread
detachment, but not keeping it attached in the first place.

It is possible to do this from user code, though ideally needs fixing
in GStreamer-Java. It depends what you're doing though. For example,
if you're using an RGBDataSink.Listener you can try adding the
following (pseudo) code

if (firstTime) {
Native.detach(false);
}

Basically you need a callback into your code in the right GStreamer
thread. If you don't already have one, you might be able to hang it
off one of the Bus listeners (whichever one gets called first). It
needs someone who knows GStreamer better to tell you if these are
called in the right thread, though if you're only seeing them at
pipeline start then it is probably the Bus messages that are causing
your issue.

It is up to you to work out when firstTime is true - will depend how
you're using it. Will work just to always call Native.detach() but
this might add overhead.

Also worth bearing in mind these are not real threads - they're proxy
Thread objects to a single native thread. They do not have the same
overhead as creating lots of threads in Java. As you've found, the
overhead can be noticeable however.

Best wishes,

Neil

[saaleb]

Hi Neil,

Thanks for you help. 

I have added Native.detach(false) to bus callback and now I see threads number reduced from 25 to 4, but this threads leaved alive even if I called Gst.deinit(),

think this behavior is not normal. 

may be I need to do something with this threads after all works done? Calling Native.detach(false) not helps as I realized it concerns only newly created threads.

If you have same issue, can you please to get me any ideas?

среда, 7 ноября 2012 г., 12:41:06 UTC+2 пользователь neilcsmith.net написал:

[Neil C Smith]

Hi,

Actually, that is expected behaviour. If you use Native.detach(false) you have to detach the thread yourself when done. Native.detach(true) in a EOS callback will probably do it. Otherwise they're left marked as alive in the VM even when they no longer exist.

The extra feature in JNA 3.5 automatically handles Native.detach(true) when a native thread dies, so will usually stop you having to worry about it.

Best wishes,

Neil

Neil C Smith
Artist : Technologist : Adviser
http://neilcsmith.net

--

You received this message because you are subscribed to the Google Groups "gstreamer-java" group.

To view this discussion on the web visit https://groups.google.com/d/msg/gstreamer-java/-/T8o1N2lFDD0J.

[Farkas Levente]

these thread should have to be destroyed when no reference found for the
listener (at least in 3.5.1).

On 11/12/2012 03:55 PM, saaleb wrote:
> Hi Neil,
>
> Thanks for you help.
> I have added Native.detach(false) to bus callback and now I see threads
> number reduced from 25 to 4, but this threads leaved alive even if I
> called Gst.deinit(),
> think this behavior is not normal.
> may be I need to do something with this threads after all works done?
> Calling Native.detach(false) not helps as I realized it concerns only
> newly created threads.
> If you have same issue, can you please to get me any ideas?
>
>

> О©╫О©╫О©╫О©╫О©╫, 7 О©╫О©╫О©╫О©╫О©╫О©╫ 2012 О©╫., 12:41:06 UTC+2 О©╫О©╫О©╫О©╫О©╫О©╫О©╫О©╫О©╫О©╫О©╫О©╫ neilcsmith.net О©╫О©╫О©╫О©╫О©╫О©╫О©╫:
>
> On 6 November 2012 10:57, saaleb <ybalt...@gmail.com <javascript:>>

> http://code.google.com/p/praxis <http://code.google.com/p/praxis>

>
> OpenEye - specialist web solutions for the cultural, education,
> charitable and local government sectors - http://openeye.info
>

> --
> You received this message because you are subscribed to the Google
> Groups "gstreamer-java" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/gstreamer-java/-/T8o1N2lFDD0J.

> To post to this group, send email to gstream...@googlegroups.com.
> To unsubscribe from this group, send email to
> gstreamer-jav...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/gstreamer-java?hl=en.

--

Levente "Si vis pacem para bellum!"

[saaleb]

AFAIK JNA 3.5.1 & GStreamer-java still not working together, am I right?

четверг, 15 ноября 2012 г., 13:37:42 UTC+2 пользователь Farkas Levente написал:

[Farkas Levente]

could you retry with the current head version and
bus.setSyncHandler(null); ?

On 11/05/2012 03:44 PM, saaleb wrote:
> Hi,
>
> During load test of GStreamer-java-based application I found a lot of
> short-lived java thread (about 30) that occurs on every GST job start.
> In case of one-two simultaneous processing it have no effect, but in
> case of ~100 jobs it have a huge overhead on threads creation(context
> switching and so on).
>
> Typical call tree of this thread:
>

> 100,0% - 3 040 �s - 1
> inv. com.sun.jna.CallbackReference$DefaultCallbackProxy.callback
> 78,3% - 2 381 �s - 2 inv. org.gstreamer.lowlevel.GTypeMapper$3.fromNative
> 76,6% - 2 329 �s - 2 inv. org.gstreamer.lowlevel.NativeObject.objectFor
> 0,4% - 11 �s - 2 inv. com.sun.jna.FromNativeContext.getTargetType
> 21,2% - 643 �s - 1 inv. org.gstreamer.Bus$2.callback
> 11,0% - 334 �s - 1 inv. org.gstreamer.$Proxy4.gst_mini_object_unref
> 7,1% - 216 �s - 1 inv. java.util.concurrent.Executor.execute
> 0,4% - 13 �s - 1 inv. org.gstreamer.Gst.getExecutor
> 0,2% - 7 �s - 1 inv. org.gstreamer.Bus.access$2
> 0,2% - 5 �s - 1 inv. org.gstreamer.BusSyncReply.intValue
> 0,2% - 5 �s - 1 inv. org.gstreamer.Bus$2$1.<init>
> 0,2% - 5 �s - 1 inv. org.gstreamer.Bus.access$0
> 0,1% - 4 �s - 1 inv. org.gstreamer.Bus$1.syncMessage

>
>
> Have anybody collides with such situation and how can I improve a
> performance of GStreamer-java part?
>
> Thanks!
>

> --
> You received this message because you are subscribed to the Google
> Groups "gstreamer-java" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/gstreamer-java/-/ZwOPOK3bMiYJ.