CommandFailed, bad state, can't recover

[Sam Halliday]

Hi all,

I'm using akka-io (and spray-io) with wandoulabs WebSockets to send messages across a network.

If I send a message of this size, without expecting an Ack:

   connection ! Tcp.Write(FrameRender(TextFrame("A" * 36040033)))

and then send a message (any size), expecting an Ack:

    connection ! Tcp.Write(FrameRender(TextFrame("hello world")), Ack)

then I get called back with a CommandFailed (containing the Tcp.Write(ByteString(...))).

That's fine, I can understand that the writes sometimes fail and I need to retry... but the problem is that no matter how many times I resend the "hello world" frame expecting an Ack, it will always fail. The connection has gotten itself into an extremely bad place.

If I reduce the size of the non-acked frame by even one byte, then everything seems stable. I'm guessing this is the boundary of some buffer somewhere.

Does anyone have any suggestions about what could be going on and how I can recover when I get the CommandFailed?

It is *extremely* difficult to debug this thanks to https://github.com/akka/akka/issues/17898 ... every time I even think about turning on logging for some of the core actors they are spitting out to toString representation of ByteStrings, which basically blows up all my tooling.

Best regards,

Sam

[Sam Halliday]

I should point out that this example is just a way that I can reliably recreate the problem on my desktop. I have seen the same symptoms in a situation where I was expecting Acks on all messages, and not sending until the last message had cleared. Wandoulabs may have been sending ping/pong responses without asking for Acks, but they are nowhere near as large as these messages.

[Sam Halliday]

This is the fuller example: you need to edit the hostname and port:

  https://github.com/fommil/simple-spray-websockets/blob/master/src/test/scala/testing/Buggy.scala#L78