Why is the cpu usage so high in a golang tcp server?

[eric.sh...@gmail.com]

I try to implement a golang tcp server, and I found the concurrency is satisfied for me, but the CPU usage is too high(concurrency is 15W+/s, but the CPU usage is about 800% in a 24 cores linux machine). At the same time, a C++ tcp server is only about 200% usage with a similar concurrency(with libevent).

The following code is the demo of golang:

func main() {
    listen, err := net.Listen("tcp", "0.0.0.0:17379")
    if err != nil {
        fmt.Errorf(err.Error())
    }
    go acceptClient(listen)
    var channel2 = make(chan bool)
    <-channel2
}

func acceptClient(listen net.Listener) {
    for {
        sock, err := listen.Accept()
        if err != nil {
            fmt.Errorf(err.Error())
        }
        tcp := sock.(*net.TCPConn)
        tcp.SetNoDelay(true)
        var channel = make(chan bool, 10)
        go read(channel, sock.(*net.TCPConn))
        go write(channel, sock.(*net.TCPConn))
    }
}

func read(channel chan bool, sock *net.TCPConn) {
    count := 0
    for {
        var buf = make([]byte, 1024)
        n, err := sock.Read(buf)
        if err != nil {
            close(channel)
            sock.CloseRead()
            return
        }
        count += n
        x := count / 58
        count = count % 58
        for i := 0; i < x; i++ {
            channel <- true
        }
   }
}

func write(channel chan bool, sock *net.TCPConn) {
    buf := []byte("+OK\r\n")
    defer func() {
        sock.CloseWrite()
        recover()
    }()
    for {
        _, ok := <-channel
        if !ok {
            return
        }
        _, writeError := sock.Write(buf)
        if writeError != nil {
            return
        }
    }
}

And I test this tcp server by the redis-benchmark with multi-clients:

redis-benchmark -h 10.100.45.2  -p 17379 -n 1000 -q script load "redis.call('set','aaa','aaa')"

I also analyzed my golang code by the pprof, it is said CPU cost a lot of time on syscall:

[Tamás Gulácsi]

2017. december 18., hétfő 13:51:26 UTC+1 időpontban eric.sh...@gmail.com a következőt írta:

I try to implement a golang tcp server, and I found the concurrency is satisfied for me, but the CPU usage is too high(concurrency is 15W+/s, but the CPU usage is about 800% in a 24 cores linux machine). At the same time, a C++ tcp server is only about 200% usage with a similar concurrency(with libevent).

The following code is the demo of golang:

func main() {
    listen, err := net.Listen("tcp", "0.0.0.0:17379")
    if err != nil {
        fmt.Errorf(err.Error())
    }
    go acceptClient(listen)
    var channel2 = make(chan bool)
    <-channel2
}

func acceptClient(listen net.Listener) {
    for {
        sock, err := listen.Accept()
        if err != nil {
            fmt.Errorf(err.Error())
        }
        tcp := sock.(*net.TCPConn)
        tcp.SetNoDelay(true)
        var channel = make(chan bool, 10)

why buffered? Who will close it?

        go read(channel, sock.(*net.TCPConn))
        go write(channel, sock.(*net.TCPConn))
    }
}

func read(channel chan bool, sock *net.TCPConn) {
    count := 0
    for {
        var buf = make([]byte, 1024)

Why allocate a new slice on every iteration?

        n, err := sock.Read(buf)

This will be slow: TCP is a streaming protocol, so this Read can read any length between 0 (! yes !) and 1024 bytes.

Use buffering (bytes.NewReader) and some kind of message separation (e.g. bufio.Scanner for a line-oriented protocol).

Next time please include the FULL svg, not only the node you think important!

[Tamás Gulácsi]

Next time please include the full pprof svg, not just the interesting node!
 

 

[Sokolov Yura]

You didn't buffer writes.
libevent does a good job at buffer managment. Go's net.Conn and os.File are unbuffered, ie they are almost "raw" file descriptors. You should use bufio.Writer and bufio.Reader, or make buffering by hands. (Just don't forget to flush bufio.Writer if response channel is empty)

[Dan Kortschak]

bufio.NewReader

[Vasiliy Tolstov]

If I have something like this, but I know that client sends fixed sized message header 48 byte and variable sized body based on data in header, and want to minimize allocations. Does I need to use bufio?

Now I'm use goleveldb util package for reusable byte slice.

19 Дек 2017 г. 11:22 пользователь "Dan Kortschak" <dan.ko...@adelaide.edu.au> написал:

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Gulácsi Tamás]

You don't need to use bufio, but you'll need buffering to decrement the count of Read calls (thus, syscalls).

Reusing []byte slices: use a sync.Pool. Or several sync.Pools for different byte slices.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

[Jesper Louis Andersen]

In addition, look at allocation. If you can keep the garbage collector constantly busy it'll use the available CPU cores in the machine. It is likely to improve the system by quite a lot.