Can I use ChannelTrafficShapingHandler to control the network read/write speed in shuffle?

[Zhaojie Niu]

Hi All:

I am trying to control the network read/write speed with ChannelTrafficShapingHandler provided by Netty.

In TransportContext.java

I modify it as below:

public TransportChannelHandler initializePipeline(
        SocketChannel channel,
        RpcHandler channelRpcHandler) {
  try {
    // added by zhaojie
    logger.info("want to try control read bandwidth on host: " + host);
    final ChannelTrafficShapingHandler channelShaping = new ChannelTrafficShapingHandler(50, 50, 1000);

    TransportChannelHandler channelHandler = createChannelHandler(channel, channelRpcHandler);

    channel.pipeline()
            .addLast("encoder", ENCODER)
            .addLast(TransportFrameDecoder.HANDLER_NAME, NettyUtils.createFrameDecoder())
            .addLast("decoder", DECODER)
            .addLast("channelTrafficShaping", channelShaping)
            .addLast("idleStateHandler", new IdleStateHandler(0, 0, conf.connectionTimeoutMs() / 1000))
            // NOTE: Chunks are currently guaranteed to be returned in the order of request, but this
            // would require more logic to guarantee if this were not part of the same event loop.
            .addLast("handler", channelHandler);

I create a ChannelTrafficShapingHandler and register it into the pipeline of the channel. I set the write and read speed as 50kb/sec in the constructor. 

Except for it, what else do I need to do?

However, it does not work. Is this idea correct? Am I missing something?   

Is there any better way ?

Thanks.

[Rogan Dawes]

This might help:

https://stackoverflow.com/questions/34326230/how-to-use-channeltrafficshapinghandler-in-netty-4

Rogan

--
You received this message because you are subscribed to the Google Groups "Netty discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netty+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/netty/a6ce7d8a-4ef7-4f53-a8e9-fe407253ade0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Zhaojie Niu]

Rogan:

Thanks. I will try it and let you know once I solve it.

在 2017年6月13日星期二 UTC+8下午7:28:37，Rogan Dawes写道：

[Rogan Dawes]

It's an interesting question, I guess.

Reading the docs for the class, I get the sense that it is implemented at quite a high level, in that it simply writes the message it gets, then waits for a suitable amount of time to pass before it will write another, based on the size of the blob.

I would have thought that it would make more sense to simply write smaller chunks more frequently, in order to have a more continuous stream, at the desired rate.

That should be achieveable by taking in a ByteBuf, then writing e.g. 1/10th of the bandwidth's bytes every 1/10th of a second until the ByteBuf is empty, then triggering channelWritabilityChanged().

And similarly, on receipt of a ByteBuf, pass on chunks of it with slices of the ByteBuf at the defined rate until the ByteBuf is consumed, then call read() again.

Rogan

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/ecbe50f6-2da1-4afc-b3bf-6513fcef3593%40googlegroups.com.

[nzje...@gmail.com]

[Zhaojie Niu]

Rogan:

Yes. you are right, original ChannelShapingHandler implement the rate control by delaying the read/write operations.

I think both ways are the same if you have large amount data to transfer. The average rates under both approaches should be similar.

在 2017年6月13日星期二 UTC+8下午9:27:21，Rogan Dawes写道：