Stats and Tracing Memory Footprint

[Stefan Georgiev]

Hi,

I am investigating the memory footprint of our gRPC-based application. While I was looking at a heap dump I found out that the most often created String (60000 objects) is "dummy.DummyService/subscribe" which is contained in:

java.lang.String @ 0xff38c8a8  dummy.DummyService/subscribe

'- fullMethodName io.grpc.internal.CensusStatsModule$ServerTracer @ 0xff38c948

   '- [0] io.grpc.StreamTracer[2] @ 0xff38c930

      '- tracers io.grpc.internal.StatsTraceContext @ 0xff38c9e8

         |- statsTraceCtx, statsTraceCtx io.grpc.netty.NettyServerStream$TransportState @ 0xff38ca10

         |- statsTraceCtx io.grpc.internal.MessageDeframer @ 0xff38ca68

         |- statsTraceCtx io.grpc.netty.NettyServerStream @ 0xff38cb38

         |- statsTraceCtx io.grpc.internal.MessageFramer @ 0xff38cb80

For reference the second most often created String (which is not related to gRPC) has 10000 instances.

This let me to the conclusion that the tracing should be disabled (because the application is not using this data), but methods like setStatsEnabled, setTracingEnabled in io.grpc.netty.NettyServerBuilder are protected.

Environment: Java 8, gRPC 1.12.0

I have few questions:

• Why are these methods protected? Are stats and tracing vital for correctly working gRPC server?
• May be it is a good idea to intern the mentioned String?
• Are there other components which can be disabled for better performance and lower memory footprint?

Thanks,

Stefan

[Kun Zhang]

[+cc Carl, Eric]

Please see responses inline.

On Thu, Aug 2, 2018 at 8:31 AM Stefan Georgiev <stefan.g...@gmail.com> wrote:

Hi,

I am investigating the memory footprint of our gRPC-based application. While I was looking at a heap dump I found out that the most often created String (60000 objects) is "dummy.DummyService/subscribe" which is contained in:

java.lang.String @ 0xff38c8a8  dummy.DummyService/subscribe

'- fullMethodName io.grpc.internal.CensusStatsModule$ServerTracer @ 0xff38c948

   '- [0] io.grpc.StreamTracer[2] @ 0xff38c930

      '- tracers io.grpc.internal.StatsTraceContext @ 0xff38c9e8

         |- statsTraceCtx, statsTraceCtx io.grpc.netty.NettyServerStream$TransportState @ 0xff38ca10

         |- statsTraceCtx io.grpc.internal.MessageDeframer @ 0xff38ca68

         |- statsTraceCtx io.grpc.netty.NettyServerStream @ 0xff38cb38

         |- statsTraceCtx io.grpc.internal.MessageFramer @ 0xff38cb80

For reference the second most often created String (which is not related to gRPC) has 10000 instances.

This let me to the conclusion that the tracing should be disabled (because the application is not using this data), but methods like setStatsEnabled, setTracingEnabled in io.grpc.netty.NettyServerBuilder are protected.

Environment: Java 8, gRPC 1.12.0

I have few questions:

• Why are these methods protected? Are stats and tracing vital for correctly working gRPC server?

Because we haven't figured out how the public API should look like. The internal knobs is just for us to benchmark internally with and without stats. We don't want to introduce public APIs immaturely.  Without an OpenCensus implementation being loaded at runtime, the stats/tracing should add minimal overhead to each RPC. If that's not the case, we will either fix it or consider making those knobs public.

RE your main complaint, the stats/tracing code doesn't create this string. It is created in NettyServerHandler when parsing the request header. They will be created even if you disabled stats/tracing.

 

• May be it is a good idea to intern the mentioned String?

Although interning won't reduce the strings being created, it will presumably allow those strings to be GC'ed sooner, which might be beneficial.

• Are there other components which can be disabled for better performance and lower memory footprint?

Thanks,

Stefan

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/90731534-81d1-455b-ae47-55dd6d741c0f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

- Kun

[Carl Mastrangelo]

It is surprising to see the method name as a String, as Netty takes care to cache it inside of the MethodDescriptor as an AsciiString.  Can you follow the inbound links to that string and see what is retaining it?  MAT let's you do this.

[Eric Anderson]

On Thu, Aug 2, 2018 at 12:30 PM 'Carl Mastrangelo' via grpc.io <grp...@googlegroups.com> wrote:

It is surprising to see the method name as a String, as Netty takes care to cache it inside of the MethodDescriptor as an AsciiString.

This is used in the server path, where Netty won't really see the MethodDescriptor; Netty is forced to create the String in order to do a MethodDescriptor lookup from the registry.

Kun, it looks like the field should just be removed; it is unused. Am I reading that right?

Aside: Interning doesn't seem like a good idea, since there are an unbounded number of these strings (they are user-controlled and arbitrary). Storing this in the HPACK table though would probably work pretty well, if we had the infrastructure to do that.

[Kun Zhang]

I sent a PR to remove the field:

https://github.com/grpc/grpc-java/pull/4731

--

- Kun