How to control quarantining of addresses

[Derek]

We are running a simple remote actor system (Akka 2.2.1) and can reproduce events that cause addresses to be quarantined:

15:39:25,672 INFO  (AppServer-akka.actor.default-dispatcher-4) [WARN] [09/11/2013 15:39:25.672] [AppServer-akka.actor.default-dispatcher-20] [Remoting] Tried to associate with unreachable remote address [akka.tcp://Publ...@10.14.10.13:2552]. Address is now quarantined, all messages to this address will be delivered to dead letters.

My question is not about why the quarantine was initiated but rather how to control the amount of time that the address is kept in quarantine?  I observe that after ~1 minute the address is removed from quarantine and the connection is re-established.  Is there a way to reduce the wait?

I see the following configuration property exists:

# This settings controls how long a system will be quarantined after

# catastrophic communication failures that result in the loss of system

# messages. Quarantining prevents communication with the remote system

# of a given UID. This function can be disabled by setting the value

# to "off".

quarantine-systems-for = 60s

We are running with the following settings (in addition to "off" have tried quarantine-systems-for 5s) but it seems to have no impact.

      akka.remote.retry-window = 5s

      akka.remote.gate-invalid-addresses-for = 5s

      akka.remote.quarantine-systems-for = off

      akka.remote.retry-gate-closed-for = 0 s

Is there some other setting we are missing here?

Thanks

[Rajiv Kurian]

I wanted to ask the same question. We have app specific protocols to handle loss of messages. It would be ideal if we could just turn the quarantine behavior off and have Akka try to send a message every time. Remote deathwatch ideally should fire when a connection is lost. Would love to hear if I can configure Akka to behave this way.

Thanks,
Rajiv

[Patrik Nordwall]

Hi Derek, 

On Wed, Sep 11, 2013 at 11:14 PM, Derek <dkn...@gmail.com> wrote:

We are running a simple remote actor system (Akka 2.2.1) and can reproduce events that cause addresses to be quarantined:

15:39:25,672 INFO  (AppServer-akka.actor.default-dispatcher-4) [WARN] [09/11/2013 15:39:25.672] [AppServer-akka.actor.default-dispatcher-20] [Remoting] Tried to associate with unreachable remote address [akka.tcp://Publ...@10.14.10.13:2552]. Address is now quarantined, all messages to this address will be delivered to dead letters.

My question is not about why the quarantine was initiated but rather how to control the amount of time that the address is kept in quarantine?  I observe that after ~1 minute the address is removed from quarantine and the connection is re-established.  Is there a way to reduce the wait?

The setting related to this exact log message "Tried to associate with unreachable remote address" is:

akka.remote.gate-invalid-addresses-for

default is 60s, 1ms is the smallest value you can use right now

Regards,

Patrik

 

I see the following configuration property exists:

# This settings controls how long a system will be quarantined after

# catastrophic communication failures that result in the loss of system

# messages. Quarantining prevents communication with the remote system

# of a given UID. This function can be disabled by setting the value

# to "off".

quarantine-systems-for = 60s

We are running with the following settings (in addition to "off" have tried quarantine-systems-for 5s) but it seems to have no impact.

      akka.remote.retry-window = 5s

      akka.remote.gate-invalid-addresses-for = 5s

      akka.remote.quarantine-systems-for = off

      akka.remote.retry-gate-closed-for = 0 s

Is there some other setting we are missing here?

Thanks

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.

--

Patrik Nordwall
Typesafe -  Reactive apps on the JVM
Twitter: @patriknw

[Derek]

Thanks, I've got that set as follows and verified via debugger the key is set:

   akka {

      remote {

         gate-invalid-addresses-for = 5s

Despite this, the quarantine period observed is still 60 s.

[Patrik Nordwall]

Ok, what is the steps to reproduce?

/Patrik

[Derek]

We have a client-server, pub-sub system implemented using remote actors (but not clustering).

When the subscriber is created it sends a message to identify itself to the publisher to register for updates.  The publisher maintains a set of subscribers to which it pushes data at a repeating interval.  Publisher and subscriber watch each other so that publisher knows to clean up a "dead" subscriber and subscriber knows to initiate a re-connect sequence if it sees the publisher go down.

I see that if I put a breakpoint in my subscriber method that is handling a data update, when it gets hit it must be stopping the keep-alive mechanism between pub & sub(?).  The subscriber then thinks the publisher is down and quarantines the address for 60 seconds even though we have our conf set to quarantine for a much shorter period (~5 seconds).

So, while I am forcibly initiating this via a breakpoint, we expect it could happen "for real" if a network hiccup occurs, etc. and don't want to be forced to quarantine for a minute if it can be avoided.

Thanks for any ideas. 

[Akka Team]

Hi Derek,

So you also set the akka.remote.gate-invalid-addresses-for to a lower interval. Hmm, that is strange, I will investigate.

Also, it is important to know why the systems get quarantined. Quarantine is usually used when the hope of correctly recovering the link and internal states is close to zero. One example if there are several system messages that could not be delivered, or the resend buffers of system message delivery got out of synch. The solution to quarantine is usually to restart one of the systems. You might try to make the quarantine interval small, and hope that the two systems are still in synch, but that is a dangerous solution.

So I would probably investigate first the reason of the quarantine, since "ordinary" network flukes are handled without involving any quarantine -- in other words, network hiccups can be survived until state gets irrecoverably compromised.

Can you give logs with DEBUG level, so I can see what happens?

-Endre

Akka Team

Typesafe - The software stack for applications that scale

Blog: letitcrash.com
Twitter: @akkateam

[Patrik Nordwall]

Endre, note in the original email and my first reply that this is not really quarantine: "Tried to associate with unreachable remote address".

Perhaps the log message is misleading.

/Patrik

[Akka Team]

Hi Patrik,

Yes, I saw that, but that alone does not give enough information. Were there errors before that? Was it a failing name lookup? Just because a remote TCP connection fails, it should not quarantine. That is why I need to figure out the cause as well.

-Endre

[Patrik Nordwall]

Great, thanks!

[Derek]

I sent Endre a copy of our log.  Everything appeared to be running normally prior to the breakpoint being hit, which interrupts the heartbeat mechanism and results in quarantine.  Thanks.

[Akka Team]

(seems like my message was sent to Derek only, resending to the list)

Hi Derek,

What you see here is remote deathwatch in action. What you did with your breakpoint is that you basically prevented any DeathWatch heartbeats to get received. From the point of view of that system, the other node disappeared. In real world the remoting would try to reestablish the connection in the background. At some point the failure detector of the remote-watcher triggers, meaning that the other system is now declared dead, and all watched ActorRefs pointing to the disappeared system will trigger the Terminated message, and the remote system gets quarantined.

Please note, that there is a fundamental difference between "network flukes" and remote deathwatch triggers. If remoting detects flakyness (disappearing connections, timeouts) it keeps trying to reestablish (according to the various settings controlling this behavior). This behavior is completely transparent for the remote-watcher itself, that operates on a higher level. However, when deathwatch declares the other system dead, it quarantines, and therefore stops communication with that remote system (until that is restarted, or the quarantine is lifted).

The point is, that usually you _do not want_ any reconnects after quarantine! Consider the general scenario:

1. Actor A in system F watches actor B on system G

2. System G disappeares

3. Remoting tries to reestablish connection between F and G

3. At some point, remote deathwatch _must_ make a decision, that the other system is considered dead.

--- Decision made here, this is a no point of return ---

4. On system F actor A gets a Terminated in the name of B

5. System F quarantines system G

Now imagine that system F does _not_ quarantine G. Now if G ever comes back, with B still alive, then the system would reach an inconsistent situation where A has received signal that actor B is Terminated, while B might send messages to actor A violating the contract that Terminated is always the final message from an actor.

So remote deathwatch trigger should be considered as a point of no return (or at least a severe event). How long you are willing to wait for a remote system to come back can be configured with "akka.remote.watch-failure-detector.acceptable-heartbeat-pause". If you think that even a 1 day disappearance is considered normal, then you should set this setting accordingly.

Summarizing the scenarios in increasing order of severity:

0. All communications normal

1. Communication is flaky, and reconnects + message dropping happen according to the remote settings (retry-gate, retry-window, etc.)

2. Communication is considered fundamentally broken and lost -- remote system is quarantined

If you want to tolerate network flakiness, you should tune your settings so that 2. happens only when the situation seems really bad (this is completely application dependent!).

You should be also aware that when quarantining happens, all existing or pending connections are closed to the quarantined system, and all pending messages are delivered to dead letters. Now after a while, quarantine is lifted, but no connection attempts will be made until there is some actual traffic! So in effect the connection between the two systems is reestablished after both:
 - the quarantine is lifted
 - _and_ there is traffic that triggers reconnect.

I recommend tuning your system configuration so that deathwatch triggers when you think it should declare a remote system dead.

I hope this helps,

-Endre

On Thu, Sep 19, 2013 at 5:41 PM, Derek <dkn...@gmail.com> wrote:

Hi Endre,

Here is my JBoss 7.1.1.Final server log.  You will see that I've started the server, let it run for a few seconds.  Everything appears to be running normally and then the following events occur:

At 10:01:59, 296 I set breakpoint within my subscriber and it gets hit.  You see the heartbeat messages stop there.  The first message logged after that is when I kill the debugger and let things flow normally again -- it tells us the publisher is unreachable.

At 10:02:32, 077 you see the message that the subscriber has now quarantined the address of the publisher.

The subscriber then attempts to reconnect to the publisher every 5 seconds but we see "tried to associated with unreachable address - Address is now quarantined..."

At 10:03:07, 238 Finally the quarantine is lifted and the subscriber reconnects with the publisher.

So, I was wrong, looks like the quarantine is lasting 30 seconds and not 60 (but when you are under fire and waiting for a reconnect 30 seconds can feel like an eternity!).  Still, 30s is much more that the 5 seconds we have it set to in the conf file (also, tried with value "off" and still got quarantine logged here).

Here's a copy of our conf:

appServer {

  akka {

    daemonic = on

    actor.provider = "akka.remote.RemoteActorRefProvider"

    jvm-exit-on-fatal-error = off

    loglevel = "DEBUG"

    remote {

     netty.tcp {

       hostname = "" // we set this programmatically

       port = 2554

       maximum-frame-size = 1048576b

      }

      retry-window = 5s

      gate-invalid-addresses-for = 5s

      quarantine-systems-for = 5s

    }

  }

}

Thanks for your assistance.

[Derek]

Hi Endre,

Thanks for the information.  It is helpful and we'll look into further tuning as you suggested.  

That aside, the initial question remains...why did the observed quarantine period - 30s - differ from what we thought we had configured with the akka.remote.quarantine-systems-for = off/5s property?

Is this property used for something different than remote death watch quarantine described here?

With clarification on that think we can put this one to bed.

-Derek

[Akka Team]

Hi Derek,

That aside, the initial question remains...why did the observed quarantine period - 30s - differ from what we thought we had configured with the akka.remote.quarantine-systems-for = off/5s property?

The question is, is there any traffic that can trigger reconnect after the quarantine elapsed? Do you have some internal timer that fires in the 30s range? If there is traffic, and reconnect does not happen after the quarantine is lifted, then that might be a bug. I am asking, because immediately before the successful association I saw this line in your logs:

10:03:07,221 WARN  [com.ft.messaging.MarketDataSubscriber] (AppServer-akka.actor.default-dispatcher-12) MarketDataSubscriber attempting to connect to akka.tcp://MarketDataPu...@192.168.212.184:2552/user/marketDataPublisher...

If I am correct, then this is some traffic that your system sends? Does similar traffic happen before? Because if not, it might be that simply this is the first message that triggers the connection attempt. Otherwise it is a bug on our side :)

 

Is this property used for something different than remote death watch quarantine described here?

No, your expectations about the setting is correct.

 

With clarification on that think we can put this one to bed.

So if you can determine if there is actual traffic after quarantine is lifted but no reconnect, then we can decide if it is a bug or not.

Thanks for following up,

-Endre

[Derek]

As soon as we see that the remote publisher has died the subscriber initiates attempts to reconnect.  Code looks like:

      context.setReceiveTimeout(reconnectInterval)

      context.actorSelection(publisherPath) ! PubSubIdentifyMessage

Currently the reconnectInterval is 5s.  You saw the messages in the log like "MarketDataSubscriber attempting to connect..." starting at 10:02:37,101.  These repeat every 5s until the publisher is allowed to receive the message (no longer quarantined?) and sends acknowledgement back to the subscriber via a message.  This acknowledgement can be see in the log at 10:03:07,238 -- "MarketDataSubscriber connected to Publisher...".  Based on these times, it took 30 seconds before the subscriber reconnect was allowed to succeed.  Assume this is because some quarantine setting is set to 30s but in our config we are setting this to 5s.

Tracing "akka.remote" config to the log shows:

SimpleConfigObject({"quarantine-systems-for":"5s","netty":{"tcp":{"port":2554,"hostname":"10.14.10.100","maximum-frame-size":"1048576b"}},"retry-window":"5s","maximum-frame-size":"1048576b","gate-invalid-addresses-for":"5s"})

On Friday, September 20, 2013 7:31:59 AM UTC-5, Akka Team wrote:

Hi Derek,

That aside, the initial question remains...why did the observed quarantine period - 30s - differ from what we thought we had configured with the akka.remote.quarantine-systems-for = off/5s property?

The question is, is there any traffic that can trigger reconnect after the quarantine elapsed? Do you have some internal timer that fires in the 30s range? If there is traffic, and reconnect does not happen after the quarantine is lifted, then that might be a bug. I am asking, because immediately before the successful association I saw this line in your logs:

10:03:07,221 WARN  [com.ft.messaging.MarketDataSubscriber] (AppServer-akka.actor.default-dispatcher-12) MarketDataSubscriber attempting to connect to akka.tcp://MarketDataPublisherSys...@192.168.212.184:2552/user/marketDataPublisher...

[Derek]

Endre,

Looks like I've solved it.  Appears the issue was with the quarantine on the publisher side - it had in turn quarantined the subscriber, so it was blocking the subscriber's attempt to reconnect.

Given that the message akka was logging from the subscriber was:

"Tried to associate with unreachable remote address [akka.tcp://MarketDataPu...@192.168.212.184:2552]. Address is now quarantined, all messages to this address will be delivered to dead letters."

This made me think the problem was on the subscriber's side--that it was trying to associate with a quarantined address.  However, what I now believe was happening is that the subscriber sent it's reconnect message (after 5s wait), the publisher blocked it b/c it had the subscriber quarantined, then in response to failing to communicate with the publisher the subscriber put the publisher back into quarantine (for 5s per our config)?  When the publisher finally removed the subscriber from quarantine the reconnect was allowed to proceed.

Thanks again,

-Derek

On Friday, September 20, 2013 7:31:59 AM UTC-5, Akka Team wrote:

Hi Derek,

That aside, the initial question remains...why did the observed quarantine period - 30s - differ from what we thought we had configured with the akka.remote.quarantine-systems-for = off/5s property?

The question is, is there any traffic that can trigger reconnect after the quarantine elapsed? Do you have some internal timer that fires in the 30s range? If there is traffic, and reconnect does not happen after the quarantine is lifted, then that might be a bug. I am asking, because immediately before the successful association I saw this line in your logs:

10:03:07,221 WARN  [com.ft.messaging.MarketDataSubscriber] (AppServer-akka.actor.default-dispatcher-12) MarketDataSubscriber attempting to connect to akka.tcp://MarketDataPublisherSys...@192.168.212.184:2552/user/marketDataPublisher...

[Endre Sándor Varga]

2013.09.20. 18:23:25 dátumon Derek <dkn...@gmail.com> írta:

> Endre,
>
> Looks like I've solved it. Appears the issue was with the quarantine on
> the publisher side - it had in turn quarantined the subscriber, so it was
> blocking the subscriber's attempt to reconnect.

Yes, because the actual connection was still there -- you just blocked heartbeat messages with a breakpoint -- and remoting tires to send a DISASSOCIATE[reason = QUARANTINE] message to the other system hoping that it might get through by some strange turn of fate. As you see, we are quite thorough in cleaning up dead associations between systems ;)

-Endre

>
> Given that the message akka was logging from the subscriber was:
>
> "Tried to associate with unreachable remote address
> [akka.tcp://MarketDataPu...@192.168.212.184:2552]. Address is now
> quarantined, all messages to this address will be delivered to dead
> letters."
>
> This made me think the problem was on the subscriber's side--that it was
> trying to associate with a quarantined address. However, what I now
> believe was happening is that the subscriber sent it's reconnect message
> (after 5s wait), the publisher blocked it b/c it had the subscriber
> quarantined, then in response to failing to communicate with the publisher
> the subscriber put the publisher back into quarantine (for 5s per our
> config)? When the publisher finally removed the subscriber from quarantine
> the reconnect was allowed to proceed.
>
> Thanks again,
>
> -Derek
>
>
> On Friday, September 20, 2013 7:31:59 AM UTC-5, Akka Team wrote:
>>
>> Hi Derek,
>>
>>
>>
>>> That aside, the initial question remains...why did the observed
>>> quarantine period - 30s - differ from what we thought we had configured

>>> with the akka.remote.quarantine-**systems-for = off/5s property?

>>>
>>
>> The question is, is there any traffic that can trigger reconnect after the
>> quarantine elapsed? Do you have some internal timer that fires in the 30s
>> range? If there is traffic, and reconnect does not happen after the
>> quarantine is lifted, then that might be a bug. I am asking, because
>> immediately before the successful association I saw this line in your logs:
>>
>> 10:03:07,221 WARN [com.ft.messaging.MarketDataSubscriber]
>> (AppServer-akka.actor.default-dispatcher-12) MarketDataSubscriber
>> attempting to connect to akka.tcp://

>> MarketDataPu...@192.168.212.184:2552/user/marketDataPublisher...

>>>> "akka.remote.watch-failure-**detector.acceptable-heartbeat-**pause". If

>>>>> actor.provider = "akka.remote.**RemoteActorRefProvider"

>>>>>>>> So you also set the akka.remote.gate-invalid-**addre**sses-for to a

>>>>>>>>>>>>> 15:39:25,672 INFO (AppServer-akka.actor.**default****-dispatcher-4)

>>>>>>>>>>>>> [WARN] [09/11/2013 15:39:25.672] [AppServer-akka.actor.default-

>>>>>>>>>>>>> ******dispatcher-20] [Remoting] Tried to associate with
>>>>>>>>>>>>> unreachable remote address [akka.tcp://Publ...@10.14.**10****
>>>>>>>>>>>>> .13:2552 <http://Publ...@10.14.10.13:2552>]. Address is now

>>>>>>>>>>>>> quarantined, all messages to this address will be delivered to dead letters.
>>>>>>>>>>>>>
>>>>>>>>>>>>> My question is not about why the quarantine was initiated but
>>>>>>>>>>>>> rather how to control the amount of time that the address is kept in
>>>>>>>>>>>>> quarantine? I observe that after ~1 minute the address is removed from
>>>>>>>>>>>>> quarantine and the connection is re-established. Is there a way to reduce
>>>>>>>>>>>>> the wait?
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> The setting related to this exact log message "Tried to
>>>>>>>>>>>> associate with unreachable remote address" is:

>>>>>>>>>>>> akka.remote.gate-invalid-**addre****sses-for

>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> default is 60s, 1ms is the smallest value you can use right now
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Patrik
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I see the following configuration property exists:
>>>>>>>>>>>>>
>>>>>>>>>>>>> # This settings controls how long a system will be
>>>>>>>>>>>>> quarantined after
>>>>>>>>>>>>> # catastrophic communication failures that result in the loss
>>>>>>>>>>>>> of system
>>>>>>>>>>>>> # messages. Quarantining prevents communication with the remote
>>>>>>>>>>>>> system
>>>>>>>>>>>>> # of a given UID. This function can be disabled by setting the
>>>>>>>>>>>>> value
>>>>>>>>>>>>> # to "off".
>>>>>>>>>>>>> quarantine-systems-for = 60s
>>>>>>>>>>>>>
>>>>>>>>>>>>> We are running with the following settings (in addition to
>>>>>>>>>>>>> "off" have tried quarantine-systems-for 5s) but it seems to have no impact.
>>>>>>>>>>>>>
>>>>>>>>>>>>> akka.remote.retry-window = 5s

>>>>>>>>>>>>> akka.remote.gate-invalid-**addre****sses-for = 5s
>>>>>>>>>>>>> akka.remote.quarantine-**systems****-for = off
>>>>>>>>>>>>> akka.remote.retry-gate-closed-******for = 0 s

>>>>>>>>>>>>>
>>>>>>>>>>>>> Is there some other setting we are missing here?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>>>>> >>>>>>>>>> Check the FAQ: http://akka.io/faq/

>>>>>>>>>>>>> >>>>>>>>>> Search the archives: https://groups.google.com/**
>>>>>>>>>>>>> grou****p/akka-user <https://groups.google.com/group/akka-user>

>>>>>>>>>>>>> ---
>>>>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>>>>> Google Groups "Akka User List" group.
>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from

>>>>>>>>>>>>> it, send an email to akka-user+...@googlegroups.com******.

>>>>>>>>>>>>> To post to this group, send email to akka...@googlegroups.com.

>>>>>>>>>>>>> Visit this group at http://groups.google.com/**group****
>>>>>>>>>>>>> /akka-user <http://groups.google.com/group/akka-user>.
>>>>>>>>>>>>> For more options, visit https://groups.google.com/**grou****
>>>>>>>>>>>>> ps/opt_out <https://groups.google.com/groups/opt_out>.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>>
>>>>>>>>>>>> Patrik Nordwall
>>>>>>>>>>>> Typesafe <http://typesafe.com/> - Reactive apps on the JVM

>>>>>>>>>>>> Twitter: @patriknw
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>>> >>>>>>>>>> Check the FAQ: http://akka.io/faq/

>>>>>>>>>>> >>>>>>>>>> Search the archives: https://groups.google.com/**grou*
>>>>>>>>>>> ***p/akka-user <https://groups.google.com/group/akka-user>

>>>>>>>>>>> ---
>>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>>> Google Groups "Akka User List" group.
>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,

>>>>>>>>>>> send an email to akka-user+...@**googlegroups.com****.

>>>>>>>>>>> To post to this group, send email to akka...@googlegroups.com.

>>>>>>>>>>> Visit this group at http://groups.google.com/**group****
>>>>>>>>>>> /akka-user <http://groups.google.com/group/akka-user>.
>>>>>>>>>>> For more options, visit https://groups.google.com/**grou****
>>>>>>>>>>> ps/opt_out <https://groups.google.com/groups/opt_out>.

>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>> >>>>>>>>>> Check the FAQ: http://akka.io/faq/

>>>>>>>>> >>>>>>>>>> Search the archives: https://groups.google.com/**grou**
>>>>>>>>> p/akka-user <https://groups.google.com/group/akka-user>

>>>>>>>>> ---
>>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>>> Groups "Akka User List" group.
>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,

>>>>>>>>> send an email to akka-user+...@**googlegroups.com**.

>>>>>>>>> To post to this group, send email to akka...@googlegroups.com.

>>>>>>>>> Visit this group at http://groups.google.com/**group**/akka-user<http://groups.google.com/group/akka-user>
>>>>>>>>> .
>>>>>>>>> For more options, visit https://groups.google.com/**grou**
>>>>>>>>> ps/opt_out <https://groups.google.com/groups/opt_out>.

>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Akka Team
>>>>>>>> Typesafe - The software stack for applications that scale
>>>>>>>> Blog: letitcrash.com
>>>>>>>> Twitter: @akkateam
>>>>>>>>
>>>>>>>> --
>>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>> >>>>>>>>>> Check the FAQ: http://akka.io/faq/

>>>>>>>> >>>>>>>>>> Search the archives: https://groups.google.com/**grou**
>>>>>>>> p/akka-user <https://groups.google.com/group/akka-user>

>>>>>>>> ---
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "Akka User List" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,

>>>>>>>> send an email to akka-user+...@**googlegroups.com**.

>>>>>>>> To post to this group, send email to akka...@googlegroups.com.

>>>>>>>> Visit this group at http://groups.google.com/**group**/akka-user<http://groups.google.com/group/akka-user>
>>>>>>>> .
>>>>>>>> For more options, visit https://groups.google.com/**grou**ps/opt_out<https://groups.google.com/groups/opt_out>
>>>>>>>> .
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Patrik Nordwall
>>>>>>> Typesafe <http://typesafe.com/> - Reactive apps on the JVM

>>>>>>> Twitter: @patriknw
>>>>>>>
>>>>>>> --
>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>> >>>>>>>>>> Check the FAQ: http://akka.io/faq/

>>>>>>> >>>>>>>>>> Search the archives: https://groups.google.com/**grou**
>>>>>>> p/akka-user <https://groups.google.com/group/akka-user>

>>>>>>> ---
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "Akka User List" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,

>>>>>>> send an email to akka-user+...@**googlegroups.com**.

>>>>>>> To post to this group, send email to akka...@googlegroups.com.

>>>>>>> Visit this group at http://groups.google.com/**group**/akka-user<http://groups.google.com/group/akka-user>
>>>>>>> .
>>>>>>> For more options, visit https://groups.google.com/**grou**ps/opt_out<https://groups.google.com/groups/opt_out>
>>>>>>> .

>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Akka Team
>>>>>> Typesafe - The software stack for applications that scale
>>>>>> Blog: letitcrash.com
>>>>>> Twitter: @akkateam
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Akka Team
>>>> Typesafe - The software stack for applications that scale
>>>> Blog: letitcrash.com
>>>> Twitter: @akkateam
>>>>
>>>>
>>>>
>>>> --
>>>> Akka Team
>>>> Typesafe - The software stack for applications that scale
>>>> Blog: letitcrash.com
>>>> Twitter: @akkateam
>>>>
>>> --
>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>> >>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
>>> ---
>>> You received this message because you are subscribed to the Google Groups
>>> "Akka User List" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an

>>> email to akka-user+...@googlegroups.com <javascript:>.
>>> To post to this group, send email to akka...@googlegroups.com<javascript:>
>>> .

>>>
>>> Visit this group at http://groups.google.com/group/akka-user.
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>
>>
>>
>> --
>> Akka Team
>> Typesafe - The software stack for applications that scale
>> Blog: letitcrash.com
>> Twitter: @akkateam
>>
>


--

Endre Varga
Software Engineer
Typesafe - Reactive Apps on the JVM
twitter: drewhk