heap memory is suddenly growing

[fransgnl]

The heap memory of my red5 server (1.0.1) may grow suddenly until it almost crashes with "out of memory" errors after reaching the 500MB limit. Running the GC manually doesn't help. Strange enough, I am testing with only 1-2 clients over RTMPT, and the HEAP growth may happen after the client is already disconnected. I use a stream generated by ffmpeg from another server. Attached a screenshot from jconsole. Normally, the HEAP is around 20Mb. The main problem is how to debug this, it happens only about once a day after doing some testing, but until now I have no clue how to reproduce it. Couldn't find any strange thing in the red5.log file, the only suspicious thing is the following warning:

WARN  o.r.s.net.rtmp.

RTMPMinaIoHandler - Exception caught Connection reset by peer

But since I sometimes kill the ffmpeg stream this warning is to be expected or? From the log file it looks like red5 is closing the stream, disconnecting the Provider and unpublish the stream which looks alright to me.

Has anybody hints how to debug this?

[fransgnl]

In the meanwhile I found tools like jmap to save the heap space at run-time and tools like jhat and IBM HeapAnalyzer to analyze the saved heap space. Now I have to wait for the problem showing up... (one of these rare moments you really want to have the problem showing up asap ;-))

Op maandag 22 april 2013 11:46:33 UTC+2 schreef fransgnl het volgende:

[pejot]

Hi,

I use VisualVm (included with java I think) and with some help http://rejeev.blogspot.co.uk/2009/04/analyzing-memory-leak-in-java.html easy enough to figure out.

Cheers,

Piotr

[fransgnl]

Thanks Piots, jvisualvm.exe is included in my jdk indeed and an additional help to monitor the heap space. "Unfortunately", everything is working smoothly now... probably has to do with the law of Murphy ;-) Anyway, when it goes wrong one day I have some tools now to hopefully find out the real course of the memory leakage.

Op maandag 22 april 2013 15:46:36 UTC+2 schreef pejot het volgende:

[Tiago Jacobs - iMDT]

When you get a memory heap dump, you can open a ticket and we can
analyze it.

For that, you should add on your JVM the following options:

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/mem_dump_$1

Regards,
Tiago

[fransgnl]

Thanks, just a short time after I added these options I was suffering from issue 336 about "Response buffer was null after encoding". So I have added files to this issue. Not sure if it has anything to do with the heap problems I have encountered last few days where none of these warnings were present in the log files. But maybe deep down in the java code the problem is related, who knows... Nevertheless, both issues are very annoying since the red5 server cannot continue streaming and has to be restarted.

Op dinsdag 23 april 2013 00:53:11 UTC+2 schreef Tiago Daniel Jacobs - iMDT het volgende:

[fransgnl]

Bingo, the heap starts to grow, no strange errors in the log files this time. Won't take long before the heap is out of bounds. Ticket can be expected soon ;-)

Op dinsdag 23 april 2013 00:53:11 UTC+2 schreef Tiago Daniel Jacobs - iMDT het volgende:

When you get a memory heap dump, you can open a ticket and we can

[Tiago Jacobs - iMDT]

If you can provide the dump will be fine, because i have sometime today.

Tiago

Em 23/04/2013 10:25, fransgnl escreveu:

[fransgnl]

Great, currently uploading the files to my test server, I will  send you an email with all the info within 10 minutes or so.


Op dinsdag 23 april 2013 15:28:23 UTC+2 schreef Tiago Daniel Jacobs - iMDT het volgende:

[Tiago Jacobs - iMDT]

Just get the head dump.

Can you describe how many users was online, and if they was using RTMP/RTMPS or RTMPT?

Tiago

--
 
---
You received this message because you are subscribed to the Google Groups "red5" group.
To unsubscribe from this group and stop receiving emails from it, send an email to red5interest...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Tiago Jacobs - iMDT]

Relax, we will take care of this.

Removed the server credentials and included reply on mailing list again.

I'm setting up the environment and let you know.

While i'm doing it, please test with clients using no RTMPT (just RTMP).

Regards,
Tiago


Em 23/04/2013 12:20, Frans Gouverne escreveu:

Doesn't reveal the log file anything suspicious?

2013/4/23 Frans Gouverne <fran...@gmail.com>

Looking at the red5.log file, there are 6 "Action publish". But it may differ from time to time. The mp4 file is about 15 minutes so this time it was about 1,5 hour in total. I have the impression that connecting and disconnecting the clients is speeding up the problem but I am not sure. Yesterday I tried for a very long time and nothing showed up, so it is really hard to say what the real cause is. I also tried with a small mp4 file publishing more than 100 times but I did not manage to let the problem show up. Really difficult to reproduce I am afraid...

2013/4/23 Tiago Jacobs - iMDT <ti...@imdt.com.br>

I'm setting up a test environment to that.

Could you describe how many times you run ffmpeg publish until the problem happens?

Tiago

[Frans Gouverne]

ok, will try later on today or otherwise tomorrow with just RTMP, no time left at this moment.

2013/4/23 Tiago Jacobs - iMDT <ti...@imdt.com.br>

Relax, we will take care of this.

Removed the server credentials and included reply on mailing list again.

I'm setting up the environment and let you know.

While i'm doing it, please test with clients using no RTMPT (just RTMP).

Regards,
Tiago


Em 23/04/2013 12:20, Frans Gouverne escreveu:

Doesn't reveal the log file anything suspicious?

2013/4/23 Frans Gouverne <fran...@gmail.com>

Looking at the red5.log file, there are 6 "Action publish". But it may differ from time to time. The mp4 file is about 15 minutes so this time it was about 1,5 hour in total. I have the impression that connecting and disconnecting the clients is speeding up the problem but I am not sure. Yesterday I tried for a very long time and nothing showed up, so it is really hard to say what the real cause is. I also tried with a small mp4 file publishing more than 100 times but I did not manage to let the problem show up. Really difficult to reproduce I am afraid...

2013/4/23 Tiago Jacobs - iMDT <ti...@imdt.com.br>

I'm setting up a test environment to that.

Could you describe how many times you run ffmpeg publish until the problem happens?

Tiago

[Tiago Jacobs - iMDT]

Ok, environment setted up, doing it:

while [ true ] ; do ffmpeg -re -i demo.mp4 -acodec copy -vcodec
copy -f flv rtmp://192.168.2.32:1935/live/oi && echo "done" || echo fail
>> /var/log/publish.log; sleep 5; done

Will go for lunch and "hopefully" have some crash later.

Tiago

[Tiago Jacobs - iMDT]

Hi, i think we figured (and fixed) the problem.

I've made a change and i'm only double-checking if it fixed.

Will commit later tonight, probably.

Tiago

[Tiago Jacobs - iMDT]

Ok, you can test with r4623.

Tiago

[Frans Gouverne]

Cool! I will try as soon as possible.

Do I need to try the latest revision r4625?

2013/4/26 Tiago Jacobs - iMDT <ti...@imdt.com.br>

Ok, you can test with r4623.

Tiago

--

[Tiago Jacobs - iMDT]

Yes, you can try 4625.

You can get the automated build from apache at:
https://builds.apache.org/job/Red5%20Trunk/166/

Tiago

[fransgnl]

Unfortunately, 4625 did not fix the problem, same problem just showed up again... I simply added 2 clients. One player page on firefox is refreshed automatically every 30 seconds. The other player on Safari (Windows) is just playing. After about 1 hour the heap starts to grow again.

Do you need the log file(s) again?


Op vrijdag 26 april 2013 15:01:06 UTC+2 schreef Tiago Daniel Jacobs - iMDT het volgende:

[Tiago Jacobs - iMDT]

Yes, we need, also log files.

Tiago

--

[Tiago Jacobs - iMDT]

Hi, please used the updated version.

I've added some useful debug messages, so we can see what's happening
better.

As you had troubles building red5 on your machine, and this week i'm
feeling with a good heart i sent you the zip and tar.gz builds.

If the problem occours again, send me dump and logs.

https://dl.dropboxusercontent.com/u/9226799/_red5_build/red5-server-1.0.2-SNAPSHOT-server.zip
https://dl.dropboxusercontent.com/u/9226799/_red5_build/red5-server-1.0.2-SNAPSHOT-server.tar.gz

Tiago

[Tiago Jacobs - iMDT]

Hi, if you find OutOfMemory problems using the new version i sent you (that's also on SVN), post memory dump and log files and we will help.

Tiago

[Tiago Jacobs - iMDT]

Fixed in trunk. Please test and if you have problems, let us know.

Tiago