gRPC with TCP transport?

[green...@gmail.com]

Hi community,

We've run some informal benchmark to compare gRPC-go's performance with other alternatives. It's apparent that gRPC could improve in this area. One option that we'd like to experiment is to see how much performance gain we would get by using TCP instead of HTTP 2.0. I understand that HTTP2.0 has been one of the core values of the gRPC projects, but it would be interesting to understand its performance implication and explore the TCP option that might fit well in many scenarios. In order to do so, I'd like to get some advice on how to replace the transport layer. Is it sufficient to replace the implementation in the google.golang.org/grpc/transport/ package? Our initial experiment indicates that some code in the call/invocation layer are coupled with HTTP 2.0 transport, e.g., the context remembers some HTTP 2.0 related status. Your advice on how to do a clean switch to TCP would be appreciated.

Below is our benchmark data - please note it's an informal benchmark, so just for people's casual reference.

Test method: use three client machines, each making 10 connections to one server and then keep making a "hello" request in a loop (no new goroutines created); both the request and response contains a 1K sized message.

gRPC-go

Max TPS: 180K, Ave. latency: 0.82ms, server CPU: 67%

Go native RPC

Max TPS: 300K, Ave. latency: 0.26ms, server CPU: 37%

Apache Thrift with Go

Max TPS: 200K, Ave. latency: 0.29ms, server CPU: 21%

Jack

[Paul Grosu]

Hi Jack,

So I believe a lot of this stuff might not yet be documented - and I think Mark Roth is working on it - so you would be relying on my memory, which I hope you don't mind :)  This will be a fun team effort.  So you can start a TCP server via the grpc_tcp_server_start method as defined here:

https://github.com/grpc/grpc/blob/master/src/core/lib/iomgr/tcp_server_posix.c#L682-L722

Keep in mind that you can create a TCP endpoint via the grpc_tcp_create method as defined here:

https://github.com/grpc/grpc/blob/master/src/core/lib/iomgr/tcp_posix.c#L467-L493

One thing to note that the endpoint is defined via the vtable, which is a grpc_endpoint_vtable struct type, where you put references of your transport functions for reading, writing, etc. as such:

static const grpc_endpoint_vtable vtable = {tcp_read,
                                            tcp_write,
                                            tcp_get_workqueue,
                                            tcp_add_to_pollset,
                                            tcp_add_to_pollset_set,
                                            tcp_shutdown,
                                            tcp_destroy,
                                            tcp_get_peer};

The above is defined on the following lines:

https://github.com/grpc/grpc/blob/master/src/core/lib/iomgr/tcp_posix.c#L458-L465

If you are unfamiliar with the endpoint definition it is in the following file:

https://github.com/grpc/grpc/blob/master/src/core/lib/iomgr/endpoint.h#L49-L62

And looks like this:

/* An endpoint caps a streaming channel between two communicating processes.
   Examples may be: a tcp socket, <stdin+stdout>, or some shared memory. */

typedef struct grpc_endpoint grpc_endpoint;
typedef struct grpc_endpoint_vtable grpc_endpoint_vtable;

struct grpc_endpoint_vtable {
  void (*read)(grpc_exec_ctx *exec_ctx, grpc_endpoint *ep,
               gpr_slice_buffer *slices, grpc_closure *cb);
  void (*write)(grpc_exec_ctx *exec_ctx, grpc_endpoint *ep,
                gpr_slice_buffer *slices, grpc_closure *cb);
  grpc_workqueue *(*get_workqueue)(grpc_endpoint *ep);
  void (*add_to_pollset)(grpc_exec_ctx *exec_ctx, grpc_endpoint *ep,
                         grpc_pollset *pollset);
  void (*add_to_pollset_set)(grpc_exec_ctx *exec_ctx, grpc_endpoint *ep,
                             grpc_pollset_set *pollset);
  void (*shutdown)(grpc_exec_ctx *exec_ctx, grpc_endpoint *ep);
  void (*destroy)(grpc_exec_ctx *exec_ctx, grpc_endpoint *ep);
  char *(*get_peer)(grpc_endpoint *ep);
};

So basically you can write your own transport functions if you prefer.  Again all of this was based on my reading of the code for some time, but if anyone thinks I misinterpreted something please correct me.

Hope it helps,
Paul

[Nicolas Noble]

Unfortunately, Jack question was about the golang version of grpc, so these don't apply here :-)

We'll get someone from the grpc-go team to reply here.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/f14ec1d8-31e7-4f71-ac8a-907065e3bad4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Paul Grosu]

Ah, thanks Nico - yeah, should have noticed that more carefully :)  The Go implementation is definitely easier to read.

[green...@gmail.com]

Thanks Paul and Nicolas.

After a closer look at grpc-go, the implementation seems more coupled with HTTP2.0 than I thought. For example, much of the code is "stream" oriented. Plus, the protocol today is pretty much wired with HTTP2 [1]. I start to wonder whether it really makes sense to implement a TCP transport for gRPC. I would appreciate your thoughts on this.

Jack

[1] https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md

[Craig Tiller]

Start with the question: "What would I change in HTTP/2 to improve QPS?"

When we asked that question, we didn't really come up with anything interesting, and so we went with HTTP/2. (To be fair, there are some things we'd probably love to change in HTTP/2, but they're not about raw QPS... I'd love to see header frames included in flow control for instance).

I suspect there's ample space in the Go implementation to improve. Our own benchmarks indicate that the C++ implementation is between 3 and 10x faster than the Go implementation, for example - and I don't believe there's much fundamental that's going on that prevents at least closing that gap.

Questions that are interesting and merit answers:

- how much time is spent:

   - between making the API call to initiate an RPC and bytes being written to the wire client side
   - between reading bytes from the wire on the server and placing bytes back on the wire server side

   - between reading bytes from the wire client side and returning control to the application

- how much time is spent doing hpack? is a faster implementation interesting?

- how many locks, allocations, and context switches are performed for each unary request? (breaking them down into the three categories listed above would aid analysis)

Armed with this data, we can start to determine where to spend time improving things.

Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/ee3f361d-d8c8-4f79-a658-e8308bd6891d%40googlegroups.com.

[Qi Zhao]

Jack,

We believe there is ample space for performance to improve (because we have not really invested time for that) and it is among our top priority next quarter. But we have been very careful on all performance related work (e.g., memory allocation, contention, etc.) since day 1. I am not sure what environment you were running the benchmark but 0.82ms sounds insanely longer than I expect. Can you share your benchmark including protobuf if convenient? We have seen some fundamental errors when external ppl tried to benchmark grpc.

Regarding adding another transport, I am not sure what code you read. We intend to encapsulate all the transport specific detail in the "transport" package and "grpc" package should be transport-agnostic. If you saw some http specific detail in the "grpc" package, please let us know.

Craig,

Given our dashboard shows the ping-pong latency of grpc-go is good and the throughput is bad, I think the low hanging fruit would be reducing the number of syscalls and contentions.

To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscribe@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/ee3f361d-d8c8-4f79-a658-e8308bd6891d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "grpc.io" group.

To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscribe@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CAAvp3oML_SdG-yhorRK%3DT8teTLtKMQeV7bJNO%2BEQsm%3DfYAX6Vg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks,

-Qi

[green...@gmail.com]

Here is our test code:

helloworld.proto

syntax = "proto3";

option java_multiple_files = true;

option java_package = "io.grpc.examples.helloworld";

option java_outer_classname = "HelloWorldProto";

package helloworld;

// The greeting service definition.

service Greeter {

  // Sends a greeting

  rpc SayHello (HelloRequest) returns (HelloReply) {}

}

// The request message containing the user's name.

message HelloRequest {

  string name = 1;

}

// The response message containing the greetings

message HelloReply {

  string message = 1;

}

main_server.go

package main

import (

"flag"

"log"

"net"

"runtime"

"sync/atomic"

"time"

pb "benchtest_V1/GRPC/helloworld/helloworld"

"golang.org/x/net/context"

"google.golang.org/grpc"

)

var totalnum int64

type server struct{}

var addr = flag.String("ip", "localhost:50052", "addr")

const (

defaultName = "rep:1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111"

)

// SayHello implements helloworld.GreeterServer

func (s *server) SayHello(ctx context.Context, in *pb.HelloRequest) (*pb.HelloReply, error) {

return &pb.HelloReply{Message: defaultName}, nil

}

func main() {

flag.Parse()

lis, err := net.Listen("tcp", *addr)

if err != nil {

log.Fatalf("failed to listen: %v", err)

}

s := grpc.NewServer()

pb.RegisterGreeterServer(s, &server{})

s.Serve(lis)

}

main_client.go

package main

import (

"log"

"os"

"os/signal"

"time"

"golang.org/x/net/context"

"flag"

"runtime"

pb "benchtest_V1/GRPC/helloworld/helloworld"

"google.golang.org/grpc"

)

const (

defaultName = "1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111"

)

var costtime, sum int64

var count = flag.Int("c", 100, "count")

var linknum = flag.Int("l", 1, "linknum")

var addr = flag.String("ip", "localhost:50052", "addr")

func LoopCallMethod(si chan int64) {

// Set up a connection to the server.

conn, err := grpc.Dial(*addr, grpc.WithInsecure())

defer conn.Close()

if err != nil {

log.Fatalf("did not connect: %v", err)

}

c := pb.NewGreeterClient(conn)

req := &pb.HelloRequest{Name: defaultName}

// warm up

for j := 0; j < 100; j++ {

_, err := c.SayHello(context.Background(), req)

if err != nil {

log.Fatalf("could not greet: %v", err)

}

}

var costtime int64

callnum := 0

for i := 0; i < *count; i++ {

begin := time.Now()

_, err := c.SayHello(context.Background(), req)

if err != nil {

log.Fatalf("could not greet: %v", err)

}

costtime += time.Since(begin).Nanoseconds()

callnum++

//output one time for every 100 invokes

if callnum == 100 {

si <- costtime

costtime = 0

callnum = 0

}

}

}

func main() {

runtime.GOMAXPROCS(0)

flag.Parse()

go fun_print()

exit := make(chan os.Signal)

signal.Notify(exit, os.Kill, os.Interrupt)

chanbuffer := make(chan int64, 10000)

ch := make(chan int)

//invoke

for i := 0; i < *linknum; i++ {

go LoopCallMethod(chanbuffer)

}

//stats groutine

go func() {

for {

select {

case <-exit:

ch <- 1

case cb := <-chanbuffer:

costtime += cb

sum += 100

}

}

}()

<-ch

}

// output info each min

func fun_print() {

for {

select {

case <-time.Tick(1000 * time.Millisecond):

if sum == 0 {

sum = 1

}

log.Printf("Call succ num: %d, Delay=%v, NumGoroutine=%v\n",

sum,

costtime/(sum*int64(time.Microsecond)),

runtime.NumGoroutine())

sum, costtime = 0, 0

}

}

}

Junjie

[green...@gmail.com]

Hi Qi,

Was there any performance improvement work done in this quarter as you indicated a couple of months ago? If not, any plan? Thanks!

Jack

Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/ee3f361d-d8c8-4f79-a658-e8308bd6891d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.

To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.

Visit this group at https://groups.google.com/group/grpc-io.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CAAvp3oML_SdG-yhorRK%3DT8teTLtKMQeV7bJNO%2BEQsm%3DfYAX6Vg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks,

-Qi