A few questions on translating a C application to Netty

[Rajiv Kurian]

Here is my use case:

i) I have a long lived TCP connection from a client to a server using my own protocol.

ii) The server never writes anything to the client.

iii) The client writes messages using my own binary protocol that has its own framing.

If I was to implement this as a single threaded C application, I would do something like this:

i) Have my client thread write directly to a circular buffer.

ii) Then on the client thread itself I would keep writing the contents of the buffer whenever it is non-empty to the socket keeping track of the position of the write.

iii) I would keep writing till I got a EAGAIN/EWOULDBLOCK. WhenI got such a status I would express interest in the writable event. In the mean time my client would keep writing to the circular buffer as long as there is space.

iv) On a writable event from the epoll loop I would drain the buffer and un-register writable interest on the socket.

v) In case my circular buffer is full, my client would decide whether to block or discard the message or take some alternative flow control action.

I am trying to translate this application to a Netty application and have a few questions:

1) I am not exactly sure how I would do this synchronous flow control in Netty. Netty channel writes are completely asynchronous as per http://netty.io/wiki/user-guide-for-5.x.html and I do not get notification of when a write would block. AFAICT Netty would have to do something like what I outlined in (iii) and (iv) itself. But unless I add a ChannelFutureListener() for my ctx.writeAndFlush(buffer) method calls I don't exactly know when my writes have gone through the TCP stack. Without this knowledge I cannot really do control flow in a straight forward way without some major callback hell. Is there a way I could get the write call to actually return a EAGAIN/EWOULDBLOCK like status? How would you recommend I do this kind of control flow? Is there any example I could look at for control flow?

2) How do I manage the write buffers? Given I want a single buffer per client, I'd like to manage my own buffers. I want my client to write directly to this buffer instead of serializing messages or any other kind of copying. From the Netty examples of writeAndFlush() it is not entirely clear what Netty does with the buffers I supply it with. I am guessing it tries to NOT do a copy of them i.e do a direct write, but if it blocks it probably will copy the part not written somewhere else? How can I prevent any extra copies here?

3) Is there a way to actually get access to the Netty's underlying NIO library so I can write my own run loop without writing a callback oriented application? I have heard that Netty has worked around a lot of NIO's deficiencies (excessive locks, allocation etc) by replacing the JDK implementation using reflection to replace the guts with better implementations. Is there a way I could use these improvements but manage my own event loop?

Thanks,

Rajiv