Etherpad failing under load

[John McLear]

Did a post about this, I can't write it here as it has lots of
pictures/graphs etc.

http://www.mclear.co.uk/2010/01/stress-testing-etherpad-open-source.html

Will be updating constantly to find out why performance is so bad! If
anyone wants to share input please do so here or in the comments on
the blog post

[mijo]

Was this "natural" load or have you generated the traffic? Was this
the number of parallel users on one pad or did everybody have his own
pad? I'm asking, because I would like to reproduce the test.

John McLear schrieb:

[K]

Were you running the same in "production" mode ?

[John McLear]

Yea I am in production made.

I got multiple people to connect (people I know on social networking sites). 

This was all load on one pad.

[Dave Walker]

John McLear wrote:
> Yea I am in production made.
>
> I got multiple people to connect (people I know on social networking
> sites).
>
> This was all load on one pad.
>
> On Sun, Jan 17, 2010 at 9:57 PM, K <es.ka...@gmail.com
> <mailto:es.ka...@gmail.com>> wrote:
>
> Were you running the same in "production" mode ?
>
> On Jan 17, 9:50 pm, John McLear <johnym...@gmail.com

> <mailto:johnym...@gmail.com>> wrote:
> > Did a post about this, I can't write it here as it has lots of
> > pictures/graphs etc.
> >
> >
> http://www.mclear.co.uk/2010/01/stress-testing-etherpad-open-source.html
> >
> > Will be updating constantly to find out why performance is so
> bad! If
> > anyone wants to share input please do so here or in the comments on
> > the blog post
>
>

I just did quite a rough experiment. I had at least 50 concurrent users
at one point, and for the users connected it performed as expected (if a
little hectic). *New* users sometimes found it hard to join, kept
timing themselves out. However i am convinced this was due to
insufficient CPU allocation.

Kind Regards,
Dave Walker

[Tiago Serafim]

Could you please share the specs of your host?

On Mon, Jan 18, 2010 at 4:51 PM, Dave Walker <DaveW...@ubuntu.com> wrote:

I just did quite a rough experiment.  I had at least 50 concurrent users at one point, and for the users connected it performed as expected (if a little hectic).  *New* users sometimes found it hard to join, kept timing themselves out.  However i am convinced this was due to insufficient CPU allocation.

Kind Regards,
Dave Walker

--
Tiago Serafim

[Dave Walker]

Hi Tiago,

It was a virtual server with a single allocated shared CPU, with ~600MB
of RAM. The RAM usage never went above ~450MB.

Kind Regards,
Dave Walker

[John McLear]

All details are on the blog post http://www.mclear.co.uk/2010/01/stress-testing-etherpad-open-source.html

We have a team in my office tonight working on this - Had a lot of
useful input from other sources (mostly java guys) that isn't on this
thread.

Hopefully once we are done I will be able to publish our findings. It
should hopefully be tonight.

On Jan 18, 7:15 pm, Dave Walker <DaveWal...@ubuntu.com> wrote:
> Tiago Serafim wrote:
> > Could you please share the specs of your host?
>

[John McLear]

Did a load more testing with dif JAVA opts, had a few discussions and
basically...

problem a) soffice.bin kicks in and goes over resources in place for
Etherpad VM
b) VMware cannot baloon quick enough evne with resources available
c) This is one heavy application..

opt specific stuff is here: http://primarypad.com/S1OgyCdAAv

shows how by changing some settings you lose usability frmo a clients
perspective..

I'm happy to share sessions with people at set times if people think
they can optimize this VM further/past my skillset (which is very
limited)..

Let me know!
Thanks

On Jan 18, 9:04 pm, John McLear <johnym...@gmail.com> wrote:
> All details are on the blog posthttp://www.mclear.co.uk/2010/01/stress-testing-etherpad-open-source.html

[John McLear]

Had some success, need to do more stress tests but it feels a lot more
stable.

PS openoffice is an issue if your server is under heavy load.

http://www.mclear.co.uk/2010/01/success-optimizing-etherpad-java-vm.html

[Andrew Fong]

How do you run Etherpad in "production" mode? I'm generally not
familiar with how Java and JVMs work.

I'm getting high CPU load when people join. It goes back down to
something sane once they're connected.

-- Andrew

[John McLear]

its in the main settings file (where you set password etc)

Are you running on a VM?  How much ram do you have?

[Andrew Fong]

I see. Thanks.

I'm running on a 1GB RAM VPS on Slicehost (not sure how they allocate
CPU -- I think it's shared). I only allocated 256M for the JVM
however.

-- Andrew

[John McLear]

have you tested under load  (more than 5 users?) i tested with 256mb of ram and it fell over way before 10 users.

[J.D. Zamfirescu]

Hi guys,

256mb is substantially below the recommended JVM heap size. You will
rapidly run out of memory at that size with the standard config
options.

If you wish to run etherpad in a very small footprint, you can try
restricting the number of worker threads with --numThreads to
something like 3 (default is 250).

Memory usage is based less on the number of users as it is based on
the number of worker threads (each of which effectively gets a large
javascript scope) and the number of pads in memory (pads get flushed
from memory periodically).

Also, as your friend said in http://primarypad.com/S1OgyCdAAv, you
definitely don't want to allocate more memory to the JVM than you
physically have available in the system, as the JVM will start
swapping and that's bad news.

J.D.

[John McLear]

Thanks for the input J.D 

I have the JVM set to 1.8 GB and the VM has 2GB allocated.

What is the user experience change when reducing the --numThreads? 

[J.D. Zamfirescu]

There might be slightly higher latency if you have a decent number of
concurrent editors.

[John McLear]

Hey JD, I'm looking for opportunities to specify the numThreads, where
would you recommend implementing this? I'm assuming this isn't an opt
I can pass to the JVM within the start up script.

On Jan 29, 9:08 pm, "J.D. Zamfirescu" <za...@google.com> wrote:
> There might be slightly higher latency if you have a decent number of
> concurrent editors.
>
>
>
> On Fri, Jan 29, 2010 at 12:37 PM, John McLear <johnym...@gmail.com> wrote:
> > Thanks for the input J.D
> > I have the JVM set to 1.8 GB and the VM has 2GB allocated.
> > What is the user experience change when reducing the --numThreads?
>
> > On Fri, Jan 29, 2010 at 8:35 PM, J.D. Zamfirescu <za...@google.com> wrote:
>
> >> Hi guys,
>
> >> 256mb is substantially below the recommended JVM heap size. You will
> >> rapidly run out of memory at that size with the standard config
> >> options.
>
> >> If you wish to run etherpad in a very small footprint, you can try
> >> restricting the number of worker threads with --numThreads to
> >> something like 3 (default is 250).
>
> >> Memory usage is based less on the number of users as it is based on
> >> the number of worker threads (each of which effectively gets a large
> >> javascript scope) and the number of pads in memory (pads get flushed
> >> from memory periodically).
>

> >> Also, as your friend said inhttp://primarypad.com/S1OgyCdAAv, you

> >> definitely don't want to allocate more memory to the JVM than you
> >> physically have available in the system, as the JVM will start
> >> swapping and that's bad news.
>
> >> J.D.
>

> >> On Fri, Jan 29, 2010 at 6:53 AM, John McLear <johnym...@gmail.com> wrote:
> >> > have you tested under load  (more than 5 users?) i tested with 256mb of
> >> > ram
> >> > and it fell over way before 10 users.
>

> >> > On Thu, Jan 28, 2010 at 7:55 PM, Andrew Fong <fongand...@gmail.com>

[J.D. Zamfirescu]

John,

It sure is an opt. Look at infrastructure/net.appjet.oui/config.scala
for all the other options you can set.

J.D.

[J.D. Zamfirescu]

Though I should probably point out that there are other options as
well, which are not specified in that file (specifically, none of the
options starting with "etherpad." are there).

J.D.

[John McLear]

@ConfigParam("Max count of worker threads.")
{ val argName = "num" }
def maxThreads = intOrElse("maxThreads", 250);

I'm assuming you mean that?

On Feb 2, 8:51 pm, "J.D. Zamfirescu" <za...@google.com> wrote:
> Though I should probably point out that there are other options as
> well, which are not specified in that file (specifically, none of the
> options starting with "etherpad." are there).
>
> J.D.
>
>
>
> On Wed, Feb 3, 2010 at 7:50 AM, J.D. Zamfirescu <za...@google.com> wrote:
> > John,
>
> > It sure is an opt. Look at infrastructure/net.appjet.oui/config.scala
> > for all the other options you can set.
>
> > J.D.
>

[John McLear]

JD, can you think of any way to simulate load without having a mass of
people available?

[J.D. Zamfirescu]

Yes, sorry, that is what I meant, maxThreads, not numThreads! :)