How to test high packet loss, netsplits, etc?

[Eax Melanhovich]

Hello.

I wanted to test that my cluster work properly under following
conditions:

* Netsplits
* High latency (i.e. 5 seconds RTT between nodes)
* Random node is dead
* High packet loss

What is a best way to solve this problem? Should I write some scripts
around Vagrant or there is a better solution?

[√iktor Ҡlang]

Hi Eax,

We use Multi Node Testing for things like this: http://doc.akka.io/docs/akka/2.3.7/dev/multi-node-testing.html

--
>>>>>>>>>>      Read the docs: http://akka.io/docs/
>>>>>>>>>>      Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>>      Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Cheers,

√

[Roland Kuhn]

Hi Eax,

you can use our test suite for inspiration, for example https://github.com/akka/akka/blob/v2.3.7/akka-cluster/src/multi-jvm/scala/akka/cluster/SurviveNetworkInstabilitySpec.scala

Regards,

Roland

Dr. Roland Kuhn
Akka Tech Lead
Typesafe – Reactive apps on the JVM.
twitter: @rolandkuhn

[Eax Melanhovich]

Victor, Roland, thanks for your quick reply.

Ok, it seems that multi node tests in Akka are able to emulate netsplits
and single node death. Is it possible to test latency and packet loss
cases too?

On Wed, 17 Dec 2014 11:23:18 +0100
Roland Kuhn <goo...@rkuhn.info> wrote:

> Hi Eax,
>
> you can use our test suite for inspiration, for example
> https://github.com/akka/akka/blob/v2.3.7/akka-cluster/src/multi-jvm/scala/akka/cluster/SurviveNetworkInstabilitySpec.scala
> <https://github.com/akka/akka/blob/v2.3.7/akka-cluster/src/multi-jvm/scala/akka/cluster/SurviveNetworkInstabilitySpec.scala>
>
> Regards,
>
> Roland
>
> > 17 dec 2014 kl. 10:21 skrev √iktor Ҡlang <viktor...@gmail.com>:
> >
> > Hi Eax,
> >
> > We use Multi Node Testing for things like this:
> > http://doc.akka.io/docs/akka/2.3.7/dev/multi-node-testing.html
> > <http://doc.akka.io/docs/akka/2.3.7/dev/multi-node-testing.html>
> >
> > On Wed, Dec 17, 2014 at 8:27 AM, Eax Melanhovich <ma...@eax.me

> > <mailto:ma...@eax.me>> wrote: Hello.

> >
> > I wanted to test that my cluster work properly under following
> > conditions:
> >
> > * Netsplits
> > * High latency (i.e. 5 seconds RTT between nodes)
> > * Random node is dead
> > * High packet loss
> >
> > What is a best way to solve this problem? Should I write some
> > scripts around Vagrant or there is a better solution?
> >
> > --
> > >>>>>>>>>> Read the docs: http://akka.io/docs/
> > >>>>>>>>>> <http://akka.io/docs/> Check the FAQ:
> > >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
> > >>>>>>>>>> <http://doc.akka.io/docs/akka/current/additional/faq.html>
> > >>>>>>>>>> Search the archives:
> > >>>>>>>>>> https://groups.google.com/group/akka-user
> > >>>>>>>>>> <https://groups.google.com/group/akka-user>
> > ---
> > You received this message because you are subscribed to the Google
> > Groups "Akka User List" group. To unsubscribe from this group and
> > stop receiving emails from it, send an email to
> > akka-user+...@googlegroups.com

> > <mailto:akka-user%2Bunsu...@googlegroups.com>. To post to this

> > group, send email to akka...@googlegroups.com

> > <mailto:akka...@googlegroups.com>. Visit this group at
> > http://groups.google.com/group/akka-user
> > <http://groups.google.com/group/akka-user>. For more options, visit
> > https://groups.google.com/d/optout
> > <https://groups.google.com/d/optout>.

> >
> >
> >
> > --
> > Cheers,
> > √
> >
> > --
> > >>>>>>>>>> Read the docs: http://akka.io/docs/

> > >>>>>>>>>> <http://akka.io/docs/> Check the FAQ:
> > >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
> > >>>>>>>>>> <http://doc.akka.io/docs/akka/current/additional/faq.html>
> > >>>>>>>>>> Search the archives:
> > >>>>>>>>>> https://groups.google.com/group/akka-user
> > >>>>>>>>>> <https://groups.google.com/group/akka-user>
> > ---
> > You received this message because you are subscribed to the Google
> > Groups "Akka User List" group. To unsubscribe from this group and
> > stop receiving emails from it, send an email to
> > akka-user+...@googlegroups.com

> > <mailto:akka-user+...@googlegroups.com>. To post to this

> > group, send email to akka...@googlegroups.com

> > <mailto:akka...@googlegroups.com>. Visit this group at
> > http://groups.google.com/group/akka-user
> > <http://groups.google.com/group/akka-user>. For more options, visit
> > https://groups.google.com/d/optout
> > <https://groups.google.com/d/optout>.

>
>
>
> Dr. Roland Kuhn
> Akka Tech Lead

> Typesafe <http://typesafe.com/> – Reactive apps on the JVM.
> twitter: @rolandkuhn
> <http://twitter.com/#!/rolandkuhn>
>

[Roland Kuhn]

Packet loss is currently not publicly exposed, see here for an example that hacks around the restriction. The reason is that this feature has not been hardened for general consumption—it is tailor-made for our internal needs. Throttling OTOH is available, see this for an example.

Regards,

Roland

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Dr. Roland Kuhn
Akka Tech Lead

Typesafe – Reactive apps on the JVM.
twitter: @rolandkuhn

[Greg Young]

Just to add to all the answers here I would normally do lab work on this with a physical cluster.

Much of this can be set in linux > 2.6

http://www.linuxfoundation.org/collaborate/workgroups/networking/netem

Cheers,

Greg

[Roland Kuhn]

17 dec 2014 kl. 18:34 skrev Greg Young <gregor...@gmail.com>:

Just to add to all the answers here I would normally do lab work on this with a physical cluster.

Of course, that’s preferable. It can be problematic to organize those facilities at home, though (says the one who failed to convince his wife of the necessity of installing a 19" rack in the cellar). And mucking around with the network settings can be a bit … unwieldy in the cloud ;-)

But yes, testing the real thing has the advantage of testing the real thing.

Regards,

Roland

[Greg Young]

Provided is fine in the cloud. I have just yet to figure out how to do
power pull testing in the cloud :)

> You received this message because you are subscribed to a topic in the

> Google Groups "Akka User List" group.

> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/akka-user/s30Si0t1kEI/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

> akka-user+...@googlegroups.com.
> To post to this group, send email to akka...@googlegroups.com.
> Visit this group at http://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.

--
Studying for the Turing test

[Greg Young]

well except of course for azure's recent crash but it doesn't happen
regularly enough to meaningful :)

[Roland Kuhn]

When I was working with big clusters at CERN we had switchable power outlets connected to the network, so I could do everything from Munich—apart from inserting a boot CD (which is why I contributed to etherboot at the time). That came in handy many times. Last I checked you could rent hardware with such conveniences at German hosting providers, but that was several years ago.

[Will Sargent]

You can also simulate network failure more directly with iptables and tc: 

https://github.com/tylertreat/Comcast

Will Sargent

Consultant, Professional Services

Typesafe, the company behind Play Framework, Akka and Scala

[Greg Young]

here is one of ours :)

http://ha.geteventstore.com/showcase/