What is up with the extremely poor C# performance throughput benchmarks?

[Peter Tiedemann]

I was looking over the benchmarks here (1.0.0, master does not seem to work):

https://performance-dot-grpc-testing.appspot.com/explore?dashboard=5712453606309888

Mostly, it seems sensible enough, C++ is fastest, Java and C# roughly tied. Then i took a look at the throughput tests, where Java shows ~10x more QPS, leaving C# closer to Python and Node.

Is there some performance issue with the C# implementation i need to be aware of?

[Carl Mastrangelo]

The reason Java is fast is because there has been a lot more time spent in making the *benchmark* fast.  Those numbers tell you what you can expect from an optimized gRPC server / client.  The core reason Java is faster is likely because there was considerable time put into profiling that benchmark code.  

[p...@configit.com]

But if the benchmark is a good indication of real world throughput, then surely there is still reason to be concerned? Or do you mean that the benchmark specific Java code (as in the benchmark application ) has been optimized?

[Carl Mastrangelo]

It is a good indication of what you can get, if you optimize your code.  I think a better idea of performance is the latency, because that tends to matter a lot more for most applications.  For that they are on par. 

[Peter Tiedemann]

But do you mean if i optimize *my* code, or if someone optimizes the *grpc* C# code base?

I am naturally concerned with how high priority .Net support has for grpc, as that is our primary platform :)

[Carl Mastrangelo]

It is if you optimize your code.  What kind of latency / throughput numbers are in your requirements?

[Nicolas Noble]

Also what's the platform requirement? I think these numbers are from Linux...

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/9d3b7f37-16b6-493b-a644-6a26f83aad35%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jan Tattermusch]

Hi,

C# performance have been improved by a lot since 1.0.0 release (from August 2016).

The results on upstream/master are temporarily broken for unrelated reasons, but normally you would see that the C# performance have been improved drastically between 1.0.0 and 1.1.0 (which is the latest release - see release notes in https://github.com/grpc/grpc/releases/tag/v1.1.0 that mentions some details about the performance improvement and how to get the best numbers).

We'll try to provide a dashboard that corresponds to the 1.1.0 release.

We are planning to bring more optimizations to C# in the future.

Jan

On Wed, Feb 8, 2017 at 9:19 AM, 'Nicolas Noble' via grpc.io <grp...@googlegroups.com> wrote:

Also what's the platform requirement? I think these numbers are from Linux...

On Wed, Feb 8, 2017, 08:53 'Carl Mastrangelo' via grpc.io <grp...@googlegroups.com> wrote:

It is if you optimize your code.  What kind of latency / throughput numbers are in your requirements?

On Wednesday, February 8, 2017 at 2:34:41 AM UTC-8, Peter Tiedemann wrote:

But do you mean if i optimize *my* code, or if someone optimizes the *grpc* C# code base?

I am naturally concerned with how high priority .Net support has for grpc, as that is our primary platform :)

On Tuesday, February 7, 2017 at 12:48:26 AM UTC+1, Carl Mastrangelo wrote:

It is a good indication of what you can get, if you optimize your code.  I think a better idea of performance is the latency, because that tends to matter a lot more for most applications.  For that they are on par. 

On Saturday, February 4, 2017 at 6:13:21 AM UTC-8, p...@configit.com wrote:

But if the benchmark is a good indication of real world throughput, then surely there is still reason to be concerned? Or do you mean that the benchmark specific Java code (as in the benchmark application ) has been optimized?

On Saturday, February 4, 2017 at 1:30:37 AM UTC+1, Carl Mastrangelo wrote:

The reason Java is fast is because there has been a lot more time spent in making the *benchmark* fast.  Those numbers tell you what you can expect from an optimized gRPC server / client.  The core reason Java is faster is likely because there was considerable time put into profiling that benchmark code.  

On Friday, February 3, 2017 at 9:57:06 AM UTC-8, Peter Tiedemann wrote:

I was looking over the benchmarks here (1.0.0, master does not seem to work):

https://performance-dot-grpc-testing.appspot.com/explore?dashboard=5712453606309888

Mostly, it seems sensible enough, C++ is fastest, Java and C# roughly tied. Then i took a look at the throughput tests, where Java shows ~10x more QPS, leaving C# closer to Python and Node.

Is there some performance issue with the C# implementation i need to be aware of?

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.

To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscribe@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/9d3b7f37-16b6-493b-a644-6a26f83aad35%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "grpc.io" group.

To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscribe@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CAOWnRi9HSq9C-75jkc7-JrTLQQ5i9GptwkR%2B-mSHKahAbyWh7g%40mail.gmail.com.

[Peter Tiedemann]

I really like what i have seen so far wrt to functionality, so i am glad to hear that performance has been improved! Mostly i was concerned that C#/.net in grpc was a bit of a "secondary" target without much support/optimization (for example, if you look at captn proto, the C# implementation is done on the side, and is a bit unsupported).

To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/9d3b7f37-16b6-493b-a644-6a26f83aad35%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.

To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

[Peter Tiedemann]

Primarily windows, but very soon also Linux via .Net Core. I guess these numbers might be based on Mono?

[Jan Tattermusch]

I think C# performance is pretty reasonable now (not quite at Java level, but much closer to it than before and C# now beats Go for example), but expect the numbers to further improve over time.

Also, if you have a performance sensitive project that uses gRPC, we'll be more than happy to hear about your experience and/or the numbers you are seeing.

To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscribe@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/92b8bdb8-a8e5-428b-adb1-b1f875daae9c%40googlegroups.com.

[Jan Tattermusch]

The numbers are based on .NET Core (CoreCLR) on Linux because that's where the future of .NET is and that's what we are trying to optimize for. Also, expect to see better numbers for Linux than Windows (Linux performance optimization is where we have the most experience & momentum).

To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscribe@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/ec7f242f-9d9d-4dac-a2f8-6f138f597956%40googlegroups.com.

[ourte...@gmail.com]

I am running into pretty severe throughput issues with both C# and C++ grpc on Windows.

I've opened an Issue here which describes what I am seeing and has sample code for both Windows and Linux:
https://github.com/grpc/grpc/issues/14378

I am seeing drastically better throughput numbers on Linux as compared with Windows running exactly the same app code.

I wonder if anyone else can help me understand what I am doing wrong or whether grpc on Windows is just not being optimized as much as on Linux?

[ourte...@gmail.com]

I've also opened another Issue on protobuf github related to C# ByteString implementation and how this is killing my throughput due to excessive garbage collection.
https://github.com/google/protobuf/issues/4206

Wonder if anyone else has run into similar obstacle...

On Friday, February 3, 2017 at 11:57:06 AM UTC-6, Peter Tiedemann wrote:

[Jan Tattermusch]

Sophia, Let's continue the discussion in the issues you filed.