netty OOM problem

[yuneng xie]

you may suffer from OOM when we are using

public static final ByteBufAllocator byteBufAllocator = UnpooledByteBufAllocator.DEFAULT;

and configure your Encoder as “preferDirect = false”, so you are using HeapBuf for encoding.

public class NettyProtocolEncoder extends MessageToByteEncoder<Object> {

    public NettyProtocolEncoder() {
        super(false);
    }

the problem is caused by the code below

protected final Object filterOutboundMessage(Object msg) {
        if (msg instanceof ByteBuf) {
            ByteBuf buf = (ByteBuf) msg;
            if (buf.isDirect()) {
                return msg;
            }

            return newDirectBuffer(buf);
        }

        if (msg instanceof FileRegion) {
            return msg;
        }

        throw new UnsupportedOperationException(
                "unsupported message type: " + StringUtil.simpleClassName(msg) + EXPECTED_TYPES);
    }

 static final class ThreadLocalDirectByteBuf extends UnpooledDirectByteBuf {

        private static final Recycler<ThreadLocalDirectByteBuf> RECYCLER = new Recycler<ThreadLocalDirectByteBuf>() {
            @Override
            protected ThreadLocalDirectByteBuf newObject(Handle handle) {
                return new ThreadLocalDirectByteBuf(handle);
            }
        };

        static ThreadLocalDirectByteBuf newInstance() {
            ThreadLocalDirectByteBuf buf = RECYCLER.get();
            buf.setRefCnt(1);
            return buf;
        }

        private final Handle handle;

        private ThreadLocalDirectByteBuf(Handle handle) {
            super(UnpooledByteBufAllocator.DEFAULT, 256, Integer.MAX_VALUE);
            this.handle = handle;
        }

        @Override
        protected void deallocate() {
            if (capacity() > THREAD_LOCAL_BUFFER_SIZE) {
                super.deallocate();
            } else {
                clear();
                RECYCLER.recycle(this, handle);
            }
        }
    }

• netty tried to convert the HeapBuf into a DirectBuf (filterOutboundMessage method)。The DirectBuf is retrieved from RECYCLER
• RECYCLER may hold a large amount of ThreadLocalDirectByteBuf.
• RECYCLER is not used to retrieve buffer when reading data. Attempts to allocate DirectBuffer may trigger FullGC
• your peer could be in the same situation. so both side get blocked.

i’m wondering why we are not using RECYCLER when reading data  ;)