akka-cluster, speeding up join/memberUp events for end-to-end tests?

[Kostas kougios]

Hi,

I am creating an end to end test for my cluster by creating 2 of them in the same jvm :

val cluster1 = Cluster(SystemName, server1Config)
val cluster2 = Cluster(SystemName, server2Config)

Thread.sleep(10000)

As you can see, I have to use a sleep() to allow the cluster to to talk to each other.

Is there a way to speed this up?

[Patrik Nordwall]

Try awaitAssert in TestKit. Something like:

    val addresses = Set(Cluster(system1).selfAddress, Cluster(system2).selfAddress)

    within(10.seconds) {

      awaitAssert {

        Cluster(system1).state.members.map(_.address) should be(addresses)

        Cluster(system1).state.members.map(_.status) should be(Set(MemberStatus.Up))

        Cluster(system2).state.members.map(_.address) should be(addresses)

        Cluster(system2).state.members.map(_.status) should be(Set(MemberStatus.Up))

      }

    }

Regards,

Patrik

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Patrik Nordwall
Typesafe -  Reactive apps on the JVM
Twitter: @patriknw

[Konstantinos Kougios]

Thanks Patrik, this worked. For me I know that 3 members should join, so I was able to do this like

TestKit.awaitCond(
   cluster1.state.members.size == 3 && cluster2.state.members.size == 3 && driverCluster.state.members.size == 3,
   20 seconds
)

Still though my test takes 11 secs to run due to some cluster initial comm delays, any ideas how to give them a kick start?

You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/_qr949I2Hxo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.

[Patrik Nordwall]

You could try to decrease the interval of these periodic tasks:

akka.cluster.gossip-interval = 500 ms

akka.cluster.leader-actions-interval = 500 ms

[Konstantinos Kougios]

Thanks Patrik but still it takes the same amount of time. While waiting I get these in the logs

[INFO] [06/22/2015 19:43:31.255] [testSystem-akka.actor.default-dispatcher-4] [akka://testSystem/deadLetters] Message [akka.cluster.InternalClusterAction$InitJoinNack] from Actor[akka://testSystem/system/cluster/core/daemon#-2049681964] to Actor[akka://testSystem/deadLetters] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [06/22/2015 19:43:32.172] [testSystem-akka.actor.default-dispatcher-3] [akka://testSystem/deadLetters] Message [akka.cluster.InternalClusterAction$InitJoin$] from Actor[akka://testSystem/system/cluster/core/daemon/firstSeedNodeProcess-1#1602924580] to Actor[akka://testSystem/deadLetters] was not delivered. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [06/22/2015 19:43:33.171] [testSystem-akka.actor.default-dispatcher-19] [akka://testSystem/deadLetters] Message [akka.cluster.InternalClusterAction$InitJoin$] from Actor[akka://testSystem/system/cluster/core/daemon/firstSeedNodeProcess-1#1602924580] to Actor[akka://testSystem/deadLetters] was not delivered. [4] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [06/22/2015 19:43:34.173] [testSystem-akka.actor.default-dispatcher-18] [akka://testSystem/deadLetters] Message [akka.cluster.InternalClusterAction$InitJoin$] from Actor[akka://testSystem/system/cluster/core/daemon/firstSeedNodeProcess-1#1602924580] to Actor[akka://testSystem/deadLetters] was not delivered. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [06/22/2015 19:43:35.173] [testSystem-akka.actor.default-dispatcher-18] [akka://testSystem/deadLetters] Message [akka.cluster.InternalClusterAction$InitJoin$] from Actor[akka://testSystem/system/cluster/core/daemon/firstSeedNodeProcess-1#1602924580] to Actor[akka://testSystem/deadLetters] was not delivered. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [06/22/2015 19:43:36.180] [testSystem-akka.actor.default-dispatcher-17] [Cluster(akka://testSystem)] Cluster Node [akka.tcp://testS...@127.0.0.1:2700] - Node [akka.tcp://testS...@127.0.0.1:2700] is JOINING, roles [index-server]

[Patrik Nordwall]

Then I think you can reduce the

akka.cluster.seed-node-timeout = 2s

It is 5 seconds by default.

I normally use the join command instead of joinSeedNodes when testing.

Cluster(system1).join(Cluster(system1).selfAddress)

Cluster(system2).join(Cluster(system1).selfAddress)

Cluster(system3).join(Cluster(system1).selfAddress)

Cheers,

Patrik

[Konstantinos Kougios]

Hi Patrik,

On 23/06/15 08:07, Patrik Nordwall wrote:

Then I think you can reduce the

akka.cluster.seed-node-timeout = 2s

it improved things, currently the overal time to startup the cluster is 7 secs (it used to be 13)

It is 5 seconds by default.

I normally use the join command instead of joinSeedNodes when testing.

Cluster(system1).join(Cluster(system1).selfAddress)

Cluster(system2).join(Cluster(system1).selfAddress)

Cluster(system3).join(Cluster(system1).selfAddress)

I tried that but it didn't work. It seems my cluster1 can see the rest, but my driverCluster can see itself. I tried doing it in different order and it seems the order I do things matters. Here is what my code looks like:

val allClusters = List(cluster1, cluster2, driverCluster)
    //    Thread.sleep(1000) <-- I tried that too, but it didn't work

for (c <- allClusters) join(c)

private def join(cluster: Cluster): Unit = for (c <- allClusters) cluster.join(c.selfAddress)


Then I am waiting like

awaitCond(
   cluster1.state.members.size == 3 && cluster2.state.members.size == 3 && driverCluster.state.members.size == 3,

   60 seconds
)

If I add some printlns to print the members of each cluster in that await, I am not getting consistent results. driverCluster seems to see itself only.

Thanks,

Kostas

[Patrik Nordwall]

You can only join an existing cluster, and bootstrap by joining one to itself.

for (c <- allClusters) c.join(cluster1.selfAddress)

[Konstantinos Kougios]

ok thanks, this now works and the test runs in 5.5 secs. With

gossip-interval = 100 ms

it goes down to 4.5s