Memory leak un Pub/Sub Emulator

[Alvaro Polo]

Hello,

I am using the Pub/Sub Emulator to perform long running tests that execute for hours, even days. The versions I am using are:

- Java: OpenJDK Runtime Environment (build 1.8.0_242-8u242-b08-1-b08)

- Google Cloud SDK: 282.0.0

The memory resources for the emulator are tight: 128MB of heap, and 128MB of internal memory (where things such as Netty buffers are stored). The point is that after many hours of activity, the emulator fails with io.netty.util.internal.OutOfDirectMemoryError exception such as:

"core-1" - "1cafdcf2-bdf9-c227-8e66-1879b4ffdd53" - "[pubsub] WARNING: An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception."
"core-1" - "1cafdcf2-bdf9-c227-8e66-1879b4ffdd53" - "[pubsub] io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 64 byte(s) of direct memory (used: 33554430, max: 33554432)"
"core-1" - "1cafdcf2-bdf9-c227-8e66-1879b4ffdd53" - "[pubsub]   at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:656)"
"core-1" - "1cafdcf2-bdf9-c227-8e66-1879b4ffdd53" - "[pubsub]   at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:611)"
"core-1" - "1cafdcf2-bdf9-c227-8e66-1879b4ffdd53" - "[pubsub]   at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)"
"core-1" - "1cafdcf2-bdf9-c227-8e66-1879b4ffdd53" - "[pubsub]   at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:66)"
"core-1" - "1cafdcf2-bdf9-c227-8e66-1879b4ffdd53" - "[pubsub]   at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.<init>(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)"
"core-1" - "1cafdcf2-bdf9-c227-8e66-1879b4ffdd53" - "[pubsub]   at io.netty.buffer.UnsafeByteBufUtil.newUnsafeDirectByteBuf(UnsafeByteBufUtil.java:625)"
"core-1" - "1cafdcf2-bdf9-c227-8e66-1879b4ffdd53" - "[pubsub]   at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:330)"
"core-1" - "1cafdcf2-bdf9-c227-8e66-1879b4ffdd53" - "[pubsub]   at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)"
"core-1" - "1cafdcf2-bdf9-c227-8e66-1879b4ffdd53" - "[pubsub]   at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)"
"core-1" - "1cafdcf2-bdf9-c227-8e66-1879b4ffdd53" - "[pubsub]   at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:139)"
"core-1" - "1cafdcf2-bdf9-c227-8e66-1879b4ffdd53" - "[pubsub]   at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)"
"core-1" - "1cafdcf2-bdf9-c227-8e66-1879b4ffdd53" - "[pubsub]   at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:147)"
"core-1" - "1cafdcf2-bdf9-c227-8e66-1879b4ffdd53" - "[pubsub]   at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:677)"
"core-1" - "1cafdcf2-bdf9-c227-8e66-1879b4ffdd53" - "[pubsub]   at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:612)"
"core-1" - "1cafdcf2-bdf9-c227-8e66-1879b4ffdd53" - "[pubsub]   at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:529)"
"core-1" - "1cafdcf2-bdf9-c227-8e66-1879b4ffdd53" - "[pubsub]   at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:491)"
"core-1" - "1cafdcf2-bdf9-c227-8e66-1879b4ffdd53" - "[pubsub]   at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)"
"core-1" - "1cafdcf2-bdf9-c227-8e66-1879b4ffdd53" - "[pubsub]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)"
"core-1" - "1cafdcf2-bdf9-c227-8e66-1879b4ffdd53" - "[pubsub]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)"
"core-1" - "1cafdcf2-bdf9-c227-8e66-1879b4ffdd53" - "[pubsub]   at java.lang.Thread.run(Thread.java:748)"

I can confirm the process instrumentation indicates the memory increases monotonically. Also the NMT reveals the same. 

The point is that even when no publication is requested to the emulator, and it just receives pull requests (obviously with zero results), the memory increases. So internal message queues apparently could not be the cause. 

I read about issues on internal memory leaks caused while using Netty. And I saw it is possible to run the Java application using the system property io.netty.leakDetection.level=PARANOID. I did that, and susprisingly Netty detects a memory leak in the emulator:

"core-1" - "95868667-a332-a6ff-7316-da06d469aaf4" - "SEVERE: LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information."
"core-1" - "95868667-a332-a6ff-7316-da06d469aaf4" - "Recent access records: "
"core-1" - "95868667-a332-a6ff-7316-da06d469aaf4" - "Created at:"
"core-1" - "95868667-a332-a6ff-7316-da06d469aaf4" - "   io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:96)"
"core-1" - "95868667-a332-a6ff-7316-da06d469aaf4" - "   io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)"
"core-1" - "95868667-a332-a6ff-7316-da06d469aaf4" - "   io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:173)"
"core-1" - "95868667-a332-a6ff-7316-da06d469aaf4" - "   io.netty.buffer.Unpooled.directBuffer(Unpooled.java:107)"
"core-1" - "95868667-a332-a6ff-7316-da06d469aaf4" - "   io.gapi.emulators.netty.HttpVersionRoutingHandler.<init>(HttpVersionRoutingHandler.java:62)"
"core-1" - "95868667-a332-a6ff-7316-da06d469aaf4" - "   io.gapi.emulators.grpc.GrpcServer$1.newHandler(GrpcServer.java:131)"
"core-1" - "95868667-a332-a6ff-7316-da06d469aaf4" - "   io.grpc.netty.NettyServerTransport.start(NettyServerTransport.java:144)"
"core-1" - "95868667-a332-a6ff-7316-da06d469aaf4" - "   io.grpc.netty.NettyServer$1.initChannel(NettyServer.java:236)"
"core-1" - "95868667-a332-a6ff-7316-da06d469aaf4" - "   io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129)"
"core-1" - "95868667-a332-a6ff-7316-da06d469aaf4" - "   io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112)"
"core-1" - "95868667-a332-a6ff-7316-da06d469aaf4" - "   io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:953)"
"core-1" - "95868667-a332-a6ff-7316-da06d469aaf4" - "   io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:610)"
"core-1" - "95868667-a332-a6ff-7316-da06d469aaf4" - "   io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46)"
"core-1" - "95868667-a332-a6ff-7316-da06d469aaf4" - "   io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1461)"
"core-1" - "95868667-a332-a6ff-7316-da06d469aaf4" - "   io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1126)"
"core-1" - "95868667-a332-a6ff-7316-da06d469aaf4" - "   io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:651)"
"core-1" - "95868667-a332-a6ff-7316-da06d469aaf4" - "   io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:515)"
"core-1" - "95868667-a332-a6ff-7316-da06d469aaf4" - "   io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:428)"
"core-1" - "95868667-a332-a6ff-7316-da06d469aaf4" - "   io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:487)"
"core-1" - "95868667-a332-a6ff-7316-da06d469aaf4" - "   io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)"
"core-1" - "95868667-a332-a6ff-7316-da06d469aaf4" - "   io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)"
"core-1" - "95868667-a332-a6ff-7316-da06d469aaf4" - "   io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:495)"
"core-1" - "95868667-a332-a6ff-7316-da06d469aaf4" - "   io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)"
"core-1" - "95868667-a332-a6ff-7316-da06d469aaf4" - "   java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)"
"core-1" - "95868667-a332-a6ff-7316-da06d469aaf4" - "   java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)"
"core-1" - "95868667-a332-a6ff-7316-da06d469aaf4" - "   java.lang.Thread.run(Thread.java:748)"

Unfortunately, I do not have access to the Pub/Sub Emulator source code to figure out what's io.gapi.emulators.netty.HttpVersionRoutingHandler doing with the memory allocation. And whether this is the root cause of the leak or not. 

Could you please provide some help here? 

Thank you in advance,

-- Alvaro

Este mensaje y cualquier archivo adjunto va dirigido exclusivamente a su destinatario, pudiendo contener información confidencial sometida a secreto profesional. No está permitida su reproducción o distribución sin la autorización expresa de Cabify. Si usted no es el destinatario final por favor elimínelo e infórmenos por esta vía.

This message and any attached file are intended exclusively for the addressee, and it may be confidential. You are not allowed to copy or disclose it without Cabify's prior written authorization. If you are not the intended recipient please delete it from your system and notify us by e-mail.

[Jun Lu]

Hi, 

Google Groups are reserved for general product discussion, StackOverflow for technical questions whereas Issue Tracker for product bugs (unexpected behaviors) and feature requests. To get a better support you should post to the relevant forum, thus please read the Community Support article for better understanding.

[Alvaro Polo]

Ups! Thank you so much. I searched for the right way to report it and I didn’t realized there is a issue tracker.

 

I just submitted the issue there.

 

Thanks for your help!

 

 

Álvaro Polo Valdenebro

Principal Software Engineer

Product Team

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/1a877c64-8754-43f7-a701-e60c7e5f64a1%40googlegroups.com.