remote communication freezes up.

[G J]

Hello,

Remote communication (that occurs about once a second) starts up fine,
and then becomes bursty, with heartbeat outages that have increasing
outage duration, and the system finally terminates/hangs with a
ClosedChannelException:null.

1. The remote machine is accessed via ssh over the public internet.

2. It is using the NATFriendlyNettyRemoteTransport.

3. When the outage occurs the internet is up, as random
pages can be accessed, including typing into another ssh
session to the remote machine.

4. Here's what's really interesting: The remote application does
a console printf, which appears fine during the outage. It's
the Akka communication over the infamous 2552 that appears to
have problems.

5. Application appears to have sufficient memory throughout, and
it is not building up unhandled messages, as I have a catch all
pattern.

Does this ring a bell?

How do I go about debugging this?

Any help is greatly appreciated.

Best Regards.

[Akka Team]

On Mon, May 21, 2012 at 4:37 AM, G J <oda...@gmail.com> wrote:
> Hello,
>
> Remote communication (that occurs about once a second) starts up fine,
> and then becomes bursty, with heartbeat outages that have increasing
> outage duration, and the system finally terminates/hangs with a
> ClosedChannelException:null.
>
> 1. The remote machine is accessed via ssh over the public internet.
>
> 2. It is using the NATFriendlyNettyRemoteTransport.

That's not standard issue man, you expect me to debug that? ;-)

>
> 3. When the outage occurs the internet is up, as random
>   pages can be accessed, including typing into another ssh
>   session to the remote machine.
>
> 4. Here's what's really interesting: The remote application does
>   a console printf, which appears fine during the outage. It's
>   the Akka communication over the infamous 2552 that appears to
>   have problems.
>
> 5. Application appears to have sufficient memory throughout, and
>   it is not building up unhandled messages, as I have a catch all
>   pattern.
>
> Does this ring a bell?

Nope

>
> How do I go about debugging this?

Enable all logging/tracing and try to find what's causing it, check
JConsole for stalled threads etc.

Cheers,
√

>
> Any help is greatly appreciated.
>
> Best Regards.
>

> --
> You received this message because you are subscribed to the Google Groups "Akka User List" group.
> To post to this group, send email to akka...@googlegroups.com.
> To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.
>



--
Akka Team
Typesafe - The software stack for applications that scale
Blog: letitcrash.com
Twitter: @akkateam

[G J]

Akka Team.

I have now turned on all Akka logging at two ends of a remote system,
and run wireshark on each end also. One end uses the NAT Friendly
Remote Provider,
and at some point messages do not come out of that end.

Here are my findings when things stop working:

-- Let (A) be the remote actor system using NAT Friendly.
-- I can send messages to (A), and wireshark on (A) says the message
has been received.
-- Akka logging on (A) also says that the message was received.
-- The message elicits a reply, and Akka logging on (A) says that a
reply is being sent.
-- However, wireshark running on (A) does not record sending that
reply,
-- and neither does wireshark on the intended receiver show anything
being received.
-- Therefore, the reply must not have been physically sent.
-- (A) is happy to and does receive however.

How can this be? (A) is receiving things fine, but failing to send
out.
This does not happen right away but after a few minutes.

Could NAT Friendly have anything to do with it?

The 10s Akka keep-alive messages are sent to (A) and ACK'd.

Any help is appreciated.

Best Regards.

> > For more options, visit this group athttp://groups.google.com/group/akka-user?hl=en.

[√iktor Ҡlang]

On Thu, May 24, 2012 at 12:34 AM, G J <oda...@gmail.com> wrote:

Akka Team.

I have now turned on all Akka logging at two ends of a remote system,
and run wireshark on each end also. One end uses the NAT Friendly
Remote Provider,
and at some point messages do not come out of that end.

Here are my findings when things stop working:

-- Let (A) be the remote actor system using NAT Friendly.
-- I can send messages to (A), and wireshark on (A) says the message
has been received.
-- Akka logging on (A) also says that the message was received.
-- The message elicits a reply, and Akka logging on (A) says that a
reply is being sent.
-- However, wireshark running on (A) does not record sending that
reply,
-- and neither does wireshark on the intended receiver show anything
being received.
-- Therefore, the reply must not have been physically sent.
-- (A) is happy to and does receive however.

How can this be? (A) is receiving things fine, but failing to send
out.
This does not happen right away but after a few minutes.

Could NAT Friendly have anything to do with it?

I wouldn't be surprised if it did. As I said earlier, I think it's too simplistic.

Cheers,

√

--
Viktor Klang

Akka Tech Lead

Typesafe - The software stack for applications that scale

Twitter: @viktorklang

[G J]

I don't know for sure that it has anything to do with NAT Friendly.
It was the easiest thing to point a finger at.

If this is completely hopeless, is there a better alternative to
implementing the remote communication myself?

Your input, as always, is appreciated.

Best Regards.

> Typesafe <http://www.typesafe.com/> - The software stack for applications
> that scale
>
> Twitter: @viktorklang

[√iktor Ҡlang]

On Thu, May 24, 2012 at 12:52 AM, G J <oda...@gmail.com> wrote:

I don't know for sure that it has anything to do with NAT Friendly.
It was the easiest thing to point a finger at.

Well, you're the first one of everyone using remoting to report this problem, and you're running on a non-official modified version of the remoting...

 

If this is completely hopeless, is there a better alternative to
implementing the remote communication myself?

Easiest is probably to check in the Akka codebase for the location that logs that your message is being sent, and then try to see what happens after that to locate what goes wrong.

Cheers,

√

--

Typesafe - The software stack for applications that scale

Twitter: @viktorklang

[G J]

Hey Viktor,

With NAT I thought there was no option but to use the non-official
version of the remote provider.
Did I miss something? I'm not tied to using it when there is a cleaner
supported path.

Thanks.

[√iktor Ҡlang]

On Thu, May 24, 2012 at 1:15 AM, G J <oda...@gmail.com> wrote:

Hey Viktor,

With NAT I thought there was no option but to use the non-official
version of the remote provider.

I didn't say you had an option, I'm just saying that I didn't write it so I cannot really support it.

 

Did I miss something? I'm not tied to using it when there is a cleaner
supported path.

What about my constructive proposal to help you find the cause?

Cheers,

√

--

Typesafe - The software stack for applications that scale

Twitter: @viktorklang

[G J]

Got it.

I'll look into tracing what the code does. Any pointers would be
appreciated.

Thanks for the help.

[G J]

Viktor,

I have a quick question. Are you aware of anywhere in the code, where
it checks to see if a write is possible, and if not, backs off for
some interval and retries.

If that were so, it would go a long way to explaining what I see:
- packets are being received ok.
- akka says it is sending a reply, but
- the reply does not make it out, until after a considerable delay.

I've downloaded the akka git source and was able to compile it, but
have not tried instrumenting or understanding it.

Best Regards.

[Roland Kuhn]

Are you looking for this?

https://github.com/akka/akka/blob/v2.0.1/akka-remote/src/main/scala/akka/remote/netty/Client.scala#L85

Regards,

Roland

Roland Kuhn
Typesafe – The software stack for applications that scale.
twitter: @rolandkuhn