Setting TCP keepalive parameters

[Mark Fine]

We are porting a Ruby EventMachine-based service and running into issues meeting requirements with go, namely adjusting TCP keepalive parameters on linux:

#define TCP_KEEPIDLE     4      /* Start keeplives after this period */
#define TCP_KEEPINTVL    5      /* Interval between keepalives */
#define TCP_KEEPCNT      6      /* Number of keepalives before death */

In Ruby EventMachine, we are able to adjust these parameters simply:

  set_sock_opt(Socket::IPPROTO_TCP, 4, X)  # TCP_KEEPIDLE
  set_sock_opt(Socket::IPPROTO_TCP, 5, Y)  # TCP_KEEPINTVL
  set_sock_opt(Socket::IPPROTO_TCP, 6, Z)  # TCP_KEEPCNT

In go, we tried getting the fd from the connection and making SetSockoptInt calls:

func setConnOpts(c *net.TCPConn) (err error) {

    c.SetKeepAlive(true)

    f, err := c.File()
    if err != nil {
        return err
    }
    defer f.Close()

    fd := int(f.Fd())

    syscall.SetsockoptInt(fd, syscall.IPPROTO_TCP, 4, X)
    syscall.SetsockoptInt(fd, syscall.IPPROTO_TCP, 5, Y)
    syscall.SetsockoptInt(fd, syscall.IPPROTO_TCP, 6, Z)

    return
}

This approach introduced poor behavior into the connection - client and server disconnect handling was significantly degraded. Trying to understand the problems introduced, we came accross the following thread, which seemed related:

http://code.google.com/p/go/issues/detail?id=2458

And looking around in the net/sockopt.go source, it seems our approach above would be problematic and that instead we would need extended APIs.

Is there a better way to get access to setting sock opts in go? Are we doing something wrong above? We would love to move off our current EventMachine-based service and on to go!

Thanks!

Mark

delete this gist

Comments are parsed with GitHub Flavored Markdown

[Kyle Lemons]

You don't say the problem you are trying to solve with keepalives.  I think you will find, however, that when you start twiddling low-level socket options, you have to twiddle a lot of them; it's possible that Ruby was doing some of this for you or had some other options already set differently.  Disconnect handling slowness bring LINGER to mind, for instance, though without knowing exactly what you mean it's hard to postulate.

[Mark Fine]

We are trying to keep track of long running TCP sockets to see if they're still alive!

 

I think you will find, however, that when you start twiddling low-level socket options, you have to twiddle a lot of them; it's possible that Ruby was doing some of this for you or had some other options already set differently.

We are interested in adjusting the keepalive parameters that we have tuned for an existing service in production - we would like to adjust a) when to start the keepalives, b) how much time between the keepalives, and c) the number of keepalives before death. Other low-level socket options that Ruby or EventMachine may or may not be twiddling should be orthogonal to the discussion around how to adjust keepalive parameters.

 

Disconnect handling slowness bring LINGER to mind, for instance, though without knowing exactly what you mean it's hard to postulate.

LINGER should be orthogonal to the change in disconnect behavior we witness simply doing f, _ := c.File(); f.Close(); on a connection. The issues I referenced seems related to that.

Hope this helps explain our context. Thanks!

Mark

[Kyle Lemons]

A minimal reproducing example showing the degraded disconnect behavior would probably be helpful in understanding the problem you're seeing.  I just tried a quick example and it didn't take any longer to tear it down with and without keepalive set.