Help tracking memory leak in Application using Gstreamer-java

[chrisb]

Hi Guys

1st of all, thanks for the gstreamer layer. 2ndly im hoping you can
help provide some guidance to aid me in removing a memory leak.

Background :

So I have a Tomcat java servlet, which on startup calls our GST.init.
Then on each subsequent request (doPost) it creates a pipeline adds
some elements then tears down the pipeline.

Obviously this is a high level view with no code.

There will be multiple requests arriving at any point in time, thus
there will be multiple pipelines and other gstuff open.

On shutting down the servlet we call deinit. However, we dont shutdown
very often, if even at all.

So after performing say 100 requests, we see large memory growth,
which appears to be in the native code. Grabbing a java dump (after
last request, and performing some garbage collection) shows that there
are a lot of references open to all sorts of gstreamer-java objects.
The actual java heap isnt that large however.

So it looks like these java references may be preventing the gstreamer
c layer from deleting memory as its reference count is likely greater
than 0.

To try and help the situation, we have a finally block which attempts
to clean up all the created elements, and we attempt to call dispose
on them. We also attempt to call finalize on the pipeline. After
setting its state to NULL and setting the reference to null. However
as stated we still have a lot of objects hanging around.

Does this seem like normal usage? I tried putting a Gst.main call in
each request thread, and then a call to quit from the Bus callbacks,
however this was resulting in the entire Gst being torn down, and
instead of just terminating 1 request we were pulling down all the
requests. So we have removed this, and currently are using await/
notityall to sync between bus callbacks and the main request thread
(tomcat exec).

In terms of versions its gstreamer-java 1.5 and JNA 3.2.5, gstreamer
java was taken from head mid september.

Any ideas guys, does this seem like a logical approach? Have anyone
else used gstreamer-java within a tomcat servlet and did they come
across anything similar ?

Thanks
Chris

[chrisb]

Hi Guys

Any ideas on this ?

Is it valid to run the API up like this, with multiple concurrent
pipelines/requests being processed concurrently?

Has anyone else done this ?

Thanks
Chris

[Andres Colubri]

As a quick note, did you try using VisualVM: http://visualvm.java.net/ ?

[chrisb]

Tried using jconsole, eclipse memory analyser, valgrind, massif, and a
few others, nothing yet.

[Matt Hurne]

I had a similar issue with a non-servlet program. I was able to use
VisualVM to discover that I needed to disconnect the various listeners
I had added (most of them on the Bus). For example, I was calling
Bus.connect(Bus.EOS) with each new Bus/pipeline I created, but was
never calling Bus.disconnect(Bus.EOS) during my cleanup routine. Bins
also have some listeners, and I'm sure there are other examples. Of
course your issue may be caused by something different altogether.
I'd definitely see what you can determine with heap dumps in VisualVM.

Matt

[chrisb]

Hi thanks for the reply.

Im using eclipse memory analyser against the heap dumps i take.

Yes you are correct and at one stage i did forget to do the
disconnects, but I have these in the cleanup now ;)

Where you using multiple concurrent pipelines ? did u have any issues
with memory growth?

[Matt Hurne]

> Hi thanks for the reply.
>
> Im using eclipse memory analyser against the heap dumps i take.
>
> Yes you are correct and at one stage i did forget to do the
> disconnects, but I have these in the cleanup now ;)
>
> Where you using multiple concurrent pipelines ? did u have any issues
> with memory growth?

Sorry, we are *not* currently using concurrent pipelines; we only have
a single pipeline running at a time. We did have an issue with memory
growth, but only due to the memory leak we caused by not disconnecting
listeners.

Matt

[Farkas Levente]

does it a bug in gstreamer-java? ie. when do you expect to automatically
remove listeners? should we've to fix something in gstreamer-java?

--
Levente "Si vis pacem para bellum!"

[Matt Hurne]

No, I don't consider it a bug in gstreamer-java. I think it is
reasonable to expect the caller of the Bus.connect() methods to be
responsible for calling the Bus.disconnect() methods. It was my
fault!

[chrisb]

Farkas, any comment on the initial posts ?

Regarding multiple concurrent pipelines ? Seen any issues with libx264
and large memory growth ?

Memory not being returned to the system ?

Thanks
Chris

On Dec 1, 1:02 pm, Matt Hurne <m...@thehurnes.com> wrote:

[Farkas Levente]

we use many 30-60 concurrent pipelines running for weeks without problems.

--

[Farkas Levente]

On 12/01/2011 02:02 PM, Matt Hurne wrote:
>> does it a bug in gstreamer-java? ie. when do you expect to automatically
>> remove listeners? should we've to fix something in gstreamer-java?
>
> No, I don't consider it a bug in gstreamer-java. I think it is
> reasonable to expect the caller of the Bus.connect() methods to be
> responsible for calling the Bus.disconnect() methods. It was my
> fault!

the real question they are disconnected when the pipeline disposed? if
not then it can be a bug.

[chrisb]

Hi Farkas

When encoder are you using?

[Farkas Levente]

when? or what or which?:-)
h264.

[chrisb]

Which ;)

Thanks, we are seeing a large memory growth, still trying to track
down the problem.

What are you doing regarding cleanup of objects? Anything specific
around buffers ? Or are you just relying on the Gstreamer-java API and
the GC to detect NULLs and then finalise ?

C

[Matt Hurne]

Not sure - we don't call dispose() on the Pipeline. We do call
setState(State.NULL) on the Pipeline. Should we be calling dispose()?
I've not seen any documentation that indicates we should. In fact,
the dispose() method doesn't have any javadoc!

[Farkas Levente]

nothing just java.
you can use visualvm to see which kind of object remain and where...

[Farkas Levente]

normally if you disconnect any listener then it should have to work or
if you nowhere refine to the pipeline and also the object which referred
gc-d then you'd have to be fine.

[chrisb]

Hi

So we are still looking at memory issue.

1) x264 threading value, causes excessive memory usage, setting
encoder threads to 1, ensuring 1 thread per encoding instance makes a
massive difference.

2) In the Java app src and sink code we have, adding a buffer.dispose
within a finally block, seems to have fixed the main leak.
Without this we rely on the GC, and when passing through large amounts
of video then this isnt scalable, so we need to forcefully release the
buffer mem there and then.

I think it would be useful to add this information into the JAVA Doc,
ie. buffers should be disposed as early as possible to avoid excessive
memory usage.

Looks like there might still be a few small leaks left, but the
buffers appears to have been the big one.

Chris

[chrisb]

Hi guys

Still looking at this leak..

From gathering various java heap dumps, we never seemed to be garbage
collecting any objects inheriting from GObject.

I tracked this down, and it looked like each object had an entry in
strongReferences map within gobject.

I have updated the disposeNativeHandle() to look like

protected void disposeNativeHandle(Pointer ptr) {
logger.log(LIFECYCLE, "Removing toggle ref " +
getClass().getSimpleName() + " (" + ptr + ")");
GOBJECT_API.g_object_remove_toggle_ref(ptr, toggle, objectID);
strongReferences.remove(this);
}

Basically added a strongReferences remove into this. And now when I
call dispose references of objects inherited from Gobject are now
garbage collected.

1) do you see any problem with this change? Functionally I see no
adverse effect, and i see far smaller java heaps.
2) im a bit concerned there are other issues in here, where we are not
gc'ing all the objects.

Any comments guys, farqas ?

I still have lots of GValueAPI$GValue references hanging about, and im
looking as to why, it looks like some link to the JNA.