Do we need to explicitly release memory for objects transferred using netty?

46 views
Skip to first unread message

Yasin Celik

unread,
Mar 20, 2017, 10:57:04 AM3/20/17
to Netty discussions

I am using netty for data transfer for a project. This part of the project is basically sending and receiving data packets. Each packet is basically an object that encapsulates some metadata and actual file chunks. When I try to transfer large files, say 5GB, somehow memory usage going up and does not free after file has been transferred. For 5gb file, memory usage starts form ~1gb and goes up to 15GB, and stays there until I quit the program.

This is my packet object:


public class BlockPacket 
{   
    private String requestId;
    private String blockId;
    private int packetSeqNo=0;
    private byte[] data;

    public BlockPacket(){
    }

    public String toString()
    {
        return "BlockPacket [requestId=" + requestId + ", blockId=" + blockId + ", packetSeqNo=" + packetSeqNo + "]";
    }

    public String getRequestId()
    {
        return requestId;
    }

    public void setRequestId(String requestId)
    {
        this.requestId = requestId;
    }

    public String getBlockId()
    {
        return blockId;
    }

    public void setBlockId(String blockId)
    {
        this.blockId = blockId;
    }

    public int getPacketSeqNo()
    {
        return packetSeqNo;
    }

    public void setPacketSeqNo(int no)
    {
        this.packetSeqNo = no;
    }

    public byte[] getData()
    {
        return data;
    }

    public void setData(byte[] data)
    {
        this.data = data;
    }
}



Here is channelRead at ServerHandler:

@Override
        public void channelRead(ChannelHandlerContext ctx, Object obj) 
        {
            if (obj instanceof BlockPacket)
            {
                BlockPacket packet = (BlockPacket) obj;
                //Do something with packet
            }else if (obj instanceof Block){
                Block block = (Block) obj;
             //Do something with block
            }else{
              //Do something else 
           }
        }


Here is my custom custom encode and decode:

@Override
        protected void encode(ChannelHandlerContext ctx, Object in, ByteBuf out) throws Exception {
            ByteArrayOutputStream outStream = new ByteArrayOutputStream();

            Output output = new Output(outStream);

            kryo.writeClassAndObject(output, in);
            output.flush();

            byte[] outArray = outStream.toByteArray();
            out.writeInt(outArray.length);
            out.writeBytes(outArray);
        }

@Override
    protected  void decode(ChannelHandlerContext ctx, ByteBuf in, List<Object> out) throws Exception {

        if (in.readableBytes() < 4)
            return;

        in.markReaderIndex();

        boolean read = false;
        if(len == 0){
            read = true;
            len = in.readInt();
        }

        if ((in.readableBytes() < len && read) || (in.readableBytes() < len+4 && !read)) {
            in.resetReaderIndex();
            read = false;
            return;
        }

        if (!read)
            in.readInt();
        byte[] buf = new byte[len];
        in.readBytes(buf);
        Input input = new Input(buf);
        Object object = kryo.readClassAndObject(input);
        out.add(object);
    }

I went through Reference counted object,
 but I could not solve my problem. Examples in this tutorial are 
generally for ByteBuf, which has release() method. However, my Packet 
object is a simple Java object. Any help is appreciated.

Rogan Dawes

unread,
Mar 20, 2017, 11:08:56 AM3/20/17
to Netty discussions
As you say, the examples in the tutorial are for ByteBuf. And your code uses ByteBuf's in one place, being the Codec. And this is where you should release the ByteBuf. ;-)

--
You received this message because you are subscribed to the Google Groups "Netty discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netty+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/netty/3fb5c8dd-2035-4a4d-bc83-f9db1a04486d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Tim Boudreau

unread,
Mar 28, 2017, 6:23:58 AM3/28/17
to Netty discussions

I am using netty for data transfer for a project. This part of the project is basically sending and receiving data packets. Each packet is basically an object that encapsulates some metadata and actual file chunks. When I try to transfer large files, say 5GB, somehow memory usage going up and does not free after file has been transferred. For 5gb file, memory usage starts form ~1gb and goes up to 15GB, and stays there until I quit the program.


From a quick look your code, it appears that the data you transfer is on the Java heap - i.e. loaded from wherever you're getting it into conventional Java objects.  So what you're seeing is pretty normal behavior from your JVM's garbage collector - memory used typically goes up until it hits the limit (which you can set with -Xmx) and stays there.  That has nothing to do with Netty - yes, you may be copying data into an off-heap buffer, but it did not start out off-heap - so 5GB of something has to pass through the Java heap on its way out the door.

If you want to test that theory, simply crank down the max heap size as far as you can and still have it work, using -Xmx when you launch the process, run your tests again and observe memory usage.  If it is smaller by the difference between the previous heap size and the new one, then it indeed has nothing to do with allocating ByteBufs.

If the data is file chunks, and you want to avoid dragging it onto the heap in the first place, see if you can find a way to use FileRegion.

If you want to see how memory is being used, JConsole can give you some info, as can -XX:+PrintGC and -XX:+PrintGCDetails - that will also answer the question of whether the allocations are on- or off-heap.

-Tim

yasinc...@gmail.com

unread,
Apr 3, 2017, 3:58:41 AM4/3/17
to Netty discussions

I am using netty for data transfer for a project. This part of the project is basically sending and receiving data packets. Each packet is basically an object that encapsulates some metadata and actual file chunks. When I try to transfer large files, say 5GB, somehow memory usage going up and does not free after file has been transferred. For 5gb file, memory usage starts form ~1gb and goes up to 15GB, and stays there until I quit the program.

This is my packet object:

Reply all
Reply to author
Forward
0 new messages