Memory leak when HttpResponseDecoder is extended

[Arnab Biswas]

Hi,

In our product,  time out functionality is added to "HttpResponseDecoder". The following is pseudo code (git gist is here). Within decode(), a timer is set and then super.decode() invoked(). But this is resulting in memory leak (Leak report below. Also the gist is here). 

If I do, "buffer.release()" at the end of decode() method it results in "io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1"

Could you please help me here?

Thanks,

Arnab

=============================================================================

public class ReadTimeoutHttpResponseDecoder extends HttpResponseDecoder {

@Override

    protected synchronized void decode(final ChannelHandlerContext ctx,

            final ByteBuf buffer, final List<Object> out) throws Exception { 

        

        if (this.timeout == null) {   

            this.timeout = this.timer.newTimeout(

                new ReadTimeoutTask(ctx), this.readTimeoutMs, 

                TimeUnit.MILLISECONDS);

        }

        log.trace("Attempting to decode message");

        super.decode(ctx, buffer, out);

        if (out == null) {

            log.trace("Still reading message");

        } else {

            log.trace("Message returned");

            cancelTimeout();

        }

    }

    @Override

    protected void decodeLast(ChannelHandlerContext ctx, ByteBuf in, List<Object> out) throws Exception {

        log.trace("Connection has been closed, not attempting to decode message");

    }

}

=============================================================================

Recent access records: 4

#4:

io.netty.buffer.AdvancedLeakAwareByteBuf.release(AdvancedLeakAwareByteBuf.java:61)

io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:252)

io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:307)

io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:293)

io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:840)

io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)

io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)

io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)

io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)

io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)

io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)

java.lang.Thread.run(Thread.java:745)

#3:

io.netty.buffer.AdvancedLeakAwareByteBuf.retain(AdvancedLeakAwareByteBuf.java:731)

io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:302)

com.abc.handler.ReadTimeoutHttpResponseDecoder.decode(ReadTimeoutHttpResponseDecoder.java:72)

io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:369)

io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:244)

io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:307)

io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:293)

io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:840)

io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)

io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)

io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)

io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)

io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)

io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)

java.lang.Thread.run(Thread.java:745)

#2:

io.netty.buffer.AdvancedLeakAwareByteBuf.readSlice(AdvancedLeakAwareByteBuf.java:113)

io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:302)

com.abc.handler.ReadTimeoutHttpResponseDecoder.decode(ReadTimeoutHttpResponseDecoder.java:72)

io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:369)

io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:244)

io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:307)

io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:293)

io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:840)

io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)

io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)

io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)

io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)

io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)

io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)

java.lang.Thread.run(Thread.java:745)

#1:

io.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:611)

io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:242)

io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)

io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)

io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)

io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)

io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)

io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)

java.lang.Thread.run(Thread.java:745)

Created at:

io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:56)

io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:177)

io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:168)

io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:129)

io.netty.channel.AdaptiveRecvByteBufAllocator$HandleImpl.allocate(AdaptiveRecvByteBufAllocator.java:104)

io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:117)

io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)

io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)

io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)

io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)

io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)

java.lang.Thread.run(Thread.java:745)

=============================================================================

[Norman Maurer]

Most likely this is because you forgot to call super.decodeLast(....).

Beside this out will never be null but may be empty if there was nothing decoded 

--
You received this message because you are subscribed to the Google Groups "Netty discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netty+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/netty/CAOG5%2B9nD2%2BkyYDQdSGDz9-GHz%2BbERcN8%2Bg8k3bVsqmFODTB8vw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Arnab Biswas]

Hi Norman,

Thanks for your input.

My mistake was even more basic. I didn't release the ByteBuf in the ultimate consumer (That happens to be the handler next to the decoder). Now that I have released it in the handler, the memory leak is no longer there.

Regarding the decodeLast(), I do NOT want to invoke super.decodeLast(). Since the goal of ReadTimeoutHttpResponseDecoder is to set a timer, I don't want to do anything during the channelInactive event. But, in that case, do I need to explicitly release the ByteBuffer within decodeLast()?

  @Override

    protected void decodeLast(ChannelHandlerContext ctx, ByteBuf in, List<Object> out) throws Exception {

        log.trace("Connection has been closed, not attempting to decode message");

        //Is this needed?

        in.release();

    }

Or is it fine if I don't release it decodeLast? My test does NOT show any memory leak even without the red code.

Thanks,

Arnab

To unsubscribe from this group and stop receiving emails from it, send an email to netty+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/CAOG5%2B9nD2%2BkyYDQdSGDz9-GHz%2BbERcN8%2Bg8k3bVsqmFODTB8vw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Netty discussions" group.

To unsubscribe from this group and stop receiving emails from it, send an email to netty+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/netty/AB57566F-4C23-4755-90E1-B31058794F48%40googlemail.com.

[Norman Maurer]

In this case you not need to call release... so all good

To unsubscribe from this group and stop receiving emails from it, send an email to netty+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/netty/CAOG5%2B9%3DdEmg55D9x7GYpDBONobDo320ZUyW91bxzBnRhyYc9eQ%40mail.gmail.com.

[Arnab Biswas]

Thank you, Norman!

Arnab

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/CAOG5%2B9%3DdEmg55D9x7GYpDBONobDo320ZUyW91bxzBnRhyYc9eQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Netty discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netty+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/7A7F7149-1B2D-4A4F-9512-198194A74361%40googlemail.com.