Memory leaks on red5 sip voice conf app (issue 736)

[Andres Ederra]

Hello BBB,

Issue 736 reports a potential memory leak when BBB runs for a long time, I've done some testing about this issue with very promising results.

My test environment was a BBB server, 5 clients and a video and audio conference going all for 10 hours.

Red5 JMV was instrumented for Java Memory analysis and performance tools where configured and ready.

Also Red5 default memory was tweaked to 384Mb (default 128M it far too low)

After the test, the clients were disconnected.

The expected result would be that JVM memory should have returned to non-load state after the test, but the result is that in my case arround 200Mb of heap memory where held, even when all clients are disconnected.
That means that a memory the leak is confirmed.

Red5-Jconsole.png attached file, shows JConsole memory graph, clearly, with moreless 250MB of heap memory used...

Ok, problem detected, now lets look for its causes.

Then I took some JVM heap dumps with jmap and analysed them with neatbeans. (If you want the heap dumps for further analysis i can send them, but they are around 200-300Mb so contact me with some uploading instructions about this if you need it).

From the heap dump analysis you can see that the culprit is:

There are many "org.bigbluebutton.voiceconf.red5.media.AudioByteData" instaces alive 200Mb worth of audio data...

Following the object references those audio packets are being held by a java.util.concurrent.LinkedBlockingQueue<AudioByteData> called audioDataQ in a instance of a org.bigbluebutton.voiceconf.red5.media.FlashToSipAudioStream.
There are several instances of FlashToSipAudioStream, all of them correct with an empty audioDataQ , but one, one instance has 500k packets on its audioDataQ queue...

Further source code analysis looks like a instance of FlashToSipAudioStream stopped consuming packets, but it is still receiving audio packets...

I don't have a deep understanding of red5 even less about the sip app, but I guess this kind of patch/hack (attached patch file, not tested, not compiled... no nothing... just an quick idea) to FlashToSipAudioStream  could do the trick? I guess a more serious solution could be implemented.

Greets

Andrés

[Richard Alam]

Thanks Andres.

This is indeed very helpful. We are trying to use jconsole too but
it's great that someone who knows how to use it has been looking for
problems also.

The java.util.concurrent.LinkedBlockingQueue<AudioByteData> holds
audio packet received from the client before sending to Asterisk/FS.
The queue is more for passing the packets from one thread to the next
which is responsible for packing them into RTP and sending to
Asterisk/FS.

I'll take a look at your patch.

Thanks again.

Richard

> --
> You received this message because you are subscribed to the Google Groups
> "BigBlueButton-dev" group.
> To post to this group, send email to bigblueb...@googlegroups.com.
> To unsubscribe from this group, send email to
> bigbluebutton-...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/bigbluebutton-dev?hl=en.
>

--
---
BigBlueButton
http://www.bigbluebutton.org
http://code.google.com/p/bigbluebutton

[Paul Wilson]

excellent Andres!

> --
> You received this message because you are subscribed to the Google
> Groups "BigBlueButton-dev" group.
> To post to this group, send email to bigblueb...@googlegroups.com
> .
> To unsubscribe from this group, send email to bigbluebutton-...@googlegroups.com
> .
> For more options, visit this group at http://groups.google.com/group/bigbluebutton-dev?hl=en
> .

> <red5-jconsole.png><FlashToSipAudioStream.java.patch>

[Andres Ederra]

Hi,

I understand... Maybe it's a threading issue, as the consuming thread is stopped but the producing thread is running.

The patch(I insist I haven't even compiled it as I don't have a working source tree... shame shame...) I sent it just adds a safeguard on the production thread so that it wont send more packets if the consumer thread is not active. Proper solution would be to fix the supposed threading problem, but I guess a little defensive programming doesn't hurt.

BTW the patch is for 0.71 sources, not the actual git sources.

I attach also 3 captures from the heap analysis...

Greets

Andrés

2010/12/1 Richard Alam <ritz...@gmail.com>

[Nadia Nouri]

Hello Andres,

How did you configure the red5 and the JMX for the BBB server in order for the Jconsol to connect to the server? i've been following the instructions in this document and didn't work? (http://www.red5tutorials.net/index.php/An_idiot%27s_guide_to_RMI/JMX_remote_connections_-_without_SSL_/_authentication)

Thanks,

Nadia

Nadia Saad Noori 

(613) 421-7486

Linkedin: Nadia Noori

“'Free software' is a matter of liberty, not price. To understand the concept, you should think of 'free' as in 'free speech,' not as in 'free beer'.” -Richard M. Stallman

[Scott Morris]

Hi Nadia,

I setup JConsole yesterday.  I needed to generate some keys and are them between Red5 and JConsole. Here is some information on what I did, however, I'm not sure why it worked but it did.

I generated a key on the red5 server.

~$ sudo keytool -genkeypair -alias jconsole -keystore /usr/share/red5/conf/keystore

Then exported it

~$ sudo keytool -export -alias jconsole -keystore /usr/share/red5/conf/keystore -file jconsole.cert

Then I imported it on the machine running JConsole and generated a key for JConsole to give to the Red5 server which I imported into the previous keystore

~$ sudo keytool -import -alias jconsole_client -keystore /usr/share/red5/conf/keystore -file jconsole_client.cert

Note that I'm only showing the Red5 server side of things, the JConsole machine's commands are the same but with a different keystore path (C:\Program Files (x86)\Java\jdk1.6.0_21\bin\.jconsoleKeyStore) and certificate name jconsole_client.cert.

Upon exchanging keys I changed the keystore location in Red5 to match that of the one I used.  In /usr/share/red5/conf/red5-common.xml is a line that says:

<property name="remoteSSLKeystore" value="${red5.config_root}/keystore.jmx"/>  

I changed it to read

<property name="remoteSSLKeystore" value="${red5.config_root}/keystore"/>

Then I ran JConsole with the following command

jconsole -J-Djavax.net.ssl.keyStore="C:\Program Files (x86)\Java\jdk1.6.0_21\bin\.jconsoleKeyStore" -J-Djavax.net.ssl.keyStorePassword=password -J-Djavax.net.ssl.trustStore="C:\Program Files (x86)\Java\jdk1.6.0_21\bin\.jconsoleKeyStore" -debug

Then connected to Red5 with the URI: service:jmx:rmi://192.168.0.238:9999/jndi/rmi://192.168.0.238:9999/red5

I hope this helps, let me know if you get it working.  I have SSL off in the /usr/share/red5/conf/red5.properties file.

Scott

[Nadia Nouri]

Hello Scott,

I tried to log and it still asking for a user name and a password even when i ran on the server side

java -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.authenticate=false 

-Dcom.sun.management.jmxremote.ssl=false 


I am not getting an error because of the ssl auth. and here is the jconsole debug output when i try to put the user name and the password for the VM. 



java.lang.SecurityException: Authentication failed! Invalid username or password
	at com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticationFailure(JMXPluggableAuthenticator.java:226)
	at com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticate(JMXPluggableAuthenticator.java:203)


	at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:232)
	at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)


	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

Caused by: javax.security.auth.login.FailedLoginException: Invalid username or password


	at com.sun.jmx.remote.security.FileLoginModule.attemptAuthentication(FileLoginModule.java:450)
	at com.sun.jmx.remote.security.FileLoginModule.login(FileLoginModule.java:296)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)


	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)


	at javax.security.auth.login.LoginContext.invoke(LoginContext.java:784)

Or this is the jconsole debug output when i don't enter user name and the password.  



ava.lang.SecurityException: Authentication failed! Credentials required
	at com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticationFailure(JMXPluggableAuthenticator.java:211)
	at com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticate(JMXPluggableAuthenticator.java:163)


	at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:232)
	at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)


	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke
     etc...
java.lang.SecurityException: Authentication failed! Credentials required


	at com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticationFailure(JMXPluggableAuthenticator.java:211)
	at com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticate(JMXPluggableAuthenticator.java:163)


     etc....

The certificate exchange went fine no problem..i changed the the host to my IP in the /etc/hosts. I tried the steps in the link you 

sent and set the keystore path on the server too beside changing the red5.properties and red5-common.xml.

I am running the recent BBB VM and trying to access it from a Windows  7 machine! 

Any thoughts?


Thanks..Nadia

[Andres Ederra]

Hi Nadia,

Well... I recognize I did a little cheating...

I also found those same problems trying to attach a remote JConsole... (I got stuck with the "Authentication failed! Credentials required" and gave up) so I did run JConsole on the server and did a local attach.

I just did a minimal tigthvncserver, and fluxbox install and run JConsole there:

To install vnc and a window manager:

sudo apt-get install tightvncserver
sudo apt-get install fluxbox

To run the vncserver...

vncserver :0

And then connect from your desktop using any vnc client...

This were my java options params (modifications to /etc/init.d/red5):

MIN_MEM="-Xms128M"
MAX_MEM="-Xmx384M"

PROFILE_OPTS="-Dcom.sun.management.jmxremote"

JAVA_OPTS="-Dpython.home=lib $MIN_MEM $MAX_MEM $PROFILE_OPTS -Dred5.root=$RED5_HOME -Dlogback.ContextSelector=org.red5.logging.LoggingContextSelector -Dcatalina.useNaming=true -Djava.security.debug=failure -cp $RED5_CLASSPATH org.red5.server.Bootstrap"


A cheap and dirty solution that leaves unresolved remote jmx, but works for my current needs...

Greets

Andrés

2010/12/2 Nadia Nouri <saad...@gmail.com>

[Andres Ederra]

Hi again...

After RTFM myself... about red5 & jmx: http://trac.red5.org/wiki/Documentation/UsersReferenceManual/Red5CoreTechnologies/07-Management#Chapter7.Management and plain jmx options http://download.oracle.com/javase/6/docs/technotes/guides/management/agent.html

Here are the final instructions for proper jmx remote monitoring for bbb.


Edit your red5 launcher(/etc/init.d/red5 in the vm version) and change the line starting with "JAVA_OPTS" with this

MIN_MEM="-Xms128M"
MAX_MEM="-Xmx384M"
PROFILE_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.access.file=$RED5_HOME/conf/access.properties -Dcom.sun.management.jmxremote.password.file=$RED5_HOME/conf/password.properties"

JAVA_OPTS="-Dpython.home=lib $MIN_MEM $MAX_MEM $PROFILE_OPTS -Dred5.root=$RED5_HOME -Dlogback.ContextSelector=org.red5.logging.LoggingContextSelector -Dcatalina.useNaming=true -Djava.security.debug=failure -cp $RED5_CLASSPATH org.red5.server.Bootstrap"

This will increase default heap size to 128M-384M instead of 128M-128M and bind the jmx listener service at 9999 port.

The jmx connection must be authenticated (red5 implementation requirement, it is not designed to work without auth) and the user and passsword to use are stored at /usr/share/red5/conf/access.properties  and /usr/share/red5/conf/password.properties, you can edit them.

password.properties file must have 600 permission and owner should be red5, so change the permissions:

sudo chown red5 /usr/share/red5/conf/password.properties
sudo chomod 600 /usr/share/red5/conf/password.properties

At last you can start JConsole(or whatever jmx app you want) and enter in JConsole gui your ip and port, and the username and password (default user//password are red5user//changeme), and you are done...

Now, remember that this configuration uses rmi protocol, that means that you better be on the same network than the server you want to profile, rmi is not easy to tunnel using ssh... or using NAT...

Enjoy...

Andrés

2010/12/2 Andres Ederra <andres...@gmail.com>

[Nadia Nouri]

Andés..Thank you very much for the help.. And I am trying to access the bbb server from the same network as the server :) And thanks for the password tip, i had some doubts about that but was't sure where to go. 

Will let you know how it goes for sure!

Thanks again...

Nadia

[Nadia Nouri]

Hello Andés,

Just to let you know that i was able to connect remotely to the BBB server (or the red5 process on the BBB server to be more specific) by following a mix of what Scott had did  (http://www.componative.com/content/controller/developer/insights/jconsole2/index.html) and using the user name and password that you mentioned (as I forgot about it :)).

Thanks again guys..it worked just fine...

Cheers,

Nadia