BigBlueButton Memory Leaks issues and fixes

[Nadia Nouri]

Hello everybody,

As the subject line indicates, I am working on detecting and "trying" to find a way for preventing memory leaks in the BigBlueButton server.
I have been noticed that this issue recently been brought to attention  (BBBHost.com: http://groups.google.com/group/bigbluebutton-dev/browse_thread/thread/a96716fc16bfa144#) , but it is not new to red5 server itself.

So I am hoping by creating this thread that we will be able to point out different sources of memory leaks and take care of them.

The BBBHost.com post had mentioned the leaks in the logs and we had experienced the same on our server. So here is some of the errors associated with memory leaks that I was able to hypothesis where the root cause and trying to find a fix for them!

1. Memory leaks caused by Red5:  red5 itself has been famous for memory leaks and it looks like after running a session, in two hours the system stars leaking and the GC cant keep up with the memory usage causing the system to crash.

• One of the leaks could be caused by IServiceCapableConnection.invoke method  (http://www.red5.org/ticket/631) and apparently it wasn't really fixed by handling meory because it was related to another class!
• org.red5.server, Client Class also could be one of the spots where Red5 or BBB is leaking, (http://www.sfr-fresh.com/unix/privat/red5-0.9.1.tar.gz:a/red5-0.9.1/doc/api/org/red5/server/package-summary.html), see Client_List

2. Memory leaks caused by the BBB apps:

From the BBB log, there has been error associated with the "o.a.c.loader.WebappClassLoader " and this is more to do with the BBB apps themselves so handling GC and releasing resource back to the system after closing a connection or closing an application (chat, pretension, etc) or leaving a conference room etc.
I found this Tomcat wiki posting a good start to help with BBB case and it helped pointing me to some fixes. (http://wiki.apache.org/tomcat/MemoryLeakProtection#A48837)

3. Memory leaks cause by the VM: BBBHost.com, can we get your input on this item?

4. Memory leaks cause by Tomcat6: (There was some errors and exceptions thrown that caused by Tomcat) still looking into that.

5. Memory leaks cause by the ActiveMQ:( There was some errors and exceptions thrown that caused by ActiveMQ) still looking into that.

This is what i was able to locate so far and hoping to patch some of the issues soon. Any help on this will be much appreciated.

Regards.
--

Nadia Noori 

“'Free software' is a matter of liberty, not price. To understand the concept, you should think of 'free' as in 'free speech,' not as in 'free beer'.” -Richard M. Stallman

[HostBBB.com]

Nadia, the errors i have seen are from o.a.c.loader.WebappClassLoader
" and appear both on a vmware instance and my production servers
running 10.04 32bit on Ec2.

regards,
Stephen
hostbbb.com

On Nov 26, 12:32 am, Nadia Nouri <saadno...@gmail.com> wrote:
> Hello everybody,
>
> As the subject line indicates, I am working on detecting and "trying" to
> find a way for preventing memory leaks in the BigBlueButton server.
> I have been noticed that this issue recently been brought to attention

>  (BBBHost.com:http://groups.google.com/group/bigbluebutton-dev/browse_thread/thread...<http://groups.google.com/group/bigbluebutton-dev/browse_thread/thread...>)

> , but it is not new to red5 server itself.
>
> So I am hoping by creating this thread that we will be able to point out
> different sources of memory leaks and take care of them.
>
> The BBBHost.com post had mentioned the leaks in the logs and we had
> experienced the same on our server. So here is some of the errors associated
> with memory leaks that I was able to hypothesis where the root cause and
> trying to find a fix for them!
>

> 1. *Memory leaks caused by Red5:  *red5 itself has been famous for memory

> leaks and it looks like after running a session, in two hours the system
> stars leaking and the GC cant keep up with the memory usage causing the
> system to crash.
>

>    - One of the leaks could be caused by IServiceCapableConnection.invoke

>    method  (http://www.red5.org/ticket/631) and apparently it wasn't really
>    fixed by handling meory because it was related to another class!

>    - org.red5.server, Client Class also could be one of the spots where Red5

>    or BBB is leaking, (

>    http://www.sfr-fresh.com/unix/privat/red5-0.9.1.tar.gz:a/red5-0.9.1/d...),
>    see Client_List
>
> *
> *2. *Memory **leaks caused by the BBB apps:
>
> *From the BBB log, there has been error associated with the "

> o.a.c.loader.WebappClassLoader " and this is more to do with the BBB apps
> themselves so handling GC and releasing resource back to the system after
> closing a connection or closing an application (chat, pretension, etc) or
> leaving a conference room etc.
> I found this Tomcat wiki posting a good start to help with BBB case and it
> helped pointing me to some fixes. (http://wiki.apache.org/tomcat/MemoryLeakProtection#A48837)
>

> 3. *Memory **leaks cause by the VM: *BBBHost.com, can we get your input on
> this item?
>
> 4. *Memory **leaks cause by Tomcat6: *(There was some errors and exceptions

> thrown that caused by Tomcat) still looking into that.
>

> 5. *Memory **leaks cause by the ActiveMQ:( *There was some errors and

> exceptions thrown that caused by ActiveMQ) still looking into that.
>
> This is what i was able to locate so far and hoping to patch some of the
> issues soon. Any help on this will be much appreciated.
>
> Regards.
> --
>

> *Nadia Noori *

[Nadia Nouri]

Hello Stephen,

thanks for your reply, just one more request..is it possible to send me some of your log file? or that would be hard? Also what is the average number of attendees for a session on your sever (just a guess), and how long they would last on average?

Also, did you have any other issues with BBB crashing or users being rejected, dropped out, etc..

Thanks,

Nadia

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.
To post to this group, send email to bigblueb...@googlegroups.com.
To unsubscribe from this group, send email to bigbluebutton-...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/bigbluebutton-dev?hl=en.

--

Nadia Saad Noori 

(613) 421-7486

Linkedin: Nadia Noori

[HostBBB.com]

Nadia, take a look at issue #736.... this was a 1hour 15min
conference with 10-12 users.... it ran good, but had periods of audio
breakup for a few users during conference...

I attached all the BBB server files to this issue... for that
conference in .zip archive.

regards,
Stephen
hostbbb.com

> > bigbluebutton-...@googlegroups.com<bigbluebutton-dev%2Bunsubscr­i...@googlegroups.com>
> > .

> > For more options, visit this group at
> >http://groups.google.com/group/bigbluebutton-dev?hl=en.
>
> --
>

> *Nadia Saad Noori *
> *(613) 421-7486*
> *Linkedin: **Nadia Noori *<http://ca.linkedin.com/pub/nadia-noori/a/874/b62>

>
> “'Free software' is a matter of liberty, not price. To understand the
> concept, you should think of 'free' as in 'free speech,' not as in 'free

> beer'.” -Richard M. Stallman- Hide quoted text -
>
> - Show quoted text -

[Nadia Nouri]

Got it, many thanks for the help..

Regards, 

Nadia

To unsubscribe from this group, send email to bigbluebutton-...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/bigbluebutton-dev?hl=en.

--

Nadia Saad Noori 

(613) 421-7486

Linkedin: Nadia Noori