net: allow setting socket options before binding and connecting

[Audrius Butkevicius]

As requested by Mikio i am raising a discussion about how https://go-review.googlesource.com/c/go/+/72810 which relates back to https://github.com/golang/go/issues/9661 .

The point Mikio is raising is as follows:

> Dial and Listen functions are complexes; Should the user-defined control function be applied to derived sockets such as DNS transport for dialing? Or should we have a separate control function for such stuff? Should we need to expose internal functionality that is now part of dialing/listening for supporting user-defined control functions?

The core problem described in issue talks about is not being able to set socket options such as SO_REUSEPORT. This is my motive for working on this.

From the side of pragmatism, that is all that is being asked here. In C, setsocketopt does not propagate down, and only affects the socket in question, and this is the goal.

Yes, the Control interface might be a bit too generic, hence why this feeling of need to propagate it down comes up.

I wouldn't mind SocketOption[] as the API, this way it's clear it's only affecting the socket in question, as that is how other languages behave.

Exposing access to the runtime network poller could be another workaround for the issue, but I haven't yet evaluated how setting options before dialing would look like, and my suspicion is that it would be quite complicated, as you'd handle blocking connect syscall yourself and so on.

Having read comments on the long standing github issue, I don't think anyone is interested for this options to propagate down to other connections, nor does there seem to be any appetite to expose the runtime poller.

I am still in favour of the Control function API, as I feel it's a bit more flexible in terms of you getting a handle on the uintptr if you really need to, even if it's confusing if the control function propagates down. I think the confusion can be cleared up with adequate documentation

[b...@benburkert.com]

Hi Audrius,

I'd like to propose a slightly different API for providing setsockopt

functionality. It's designed with with a few points in mind, summarized from

the feedback on issue #9661 and my own thoughts:

* Keep the additions to the net package to a minimal.

* Avoid exposing the 'fd uintptr'-ness of sockets in the net package.

* Facilitate a way for the API to evolve outside of the standard library.

* Use packages that are already platform specific to provide platform specific

  socket option configuration.

* Allow for fine-tuning socket options without having to resort to custom

  Control funcs, long func calls, or slices of option decorators.

* Some roughness is OK if it avoids ambiguity about when to use this API. It is

  not a better net.Listen.

Here is the proposed API:

  // package net

  func ListenSyscall(sock syscall.Socket) (net.Listener, error)

  // package syscall

  type Socket interface {

    Addr() Sockaddr

    Control(f func(fd uintptr)) error

  }

  // package x/net/socket or x/sys/unix/socket

  type Config struct {

    ReusePort bool

    FastOpen  bool

    // ...

  }

  func (c Config) Open(network, addr string) (syscall.Socket, error)

Here's an example of how the API could be used in a web server:

https://play.golang.org/p/-ZcRL6MzUX

I'm interested to hear what others think.

-Ben

[Audrius Butkevicius]

Listen is not the only problem.

Dial is also the problem, and it's actually a problem that is harder to solve than listening, as there is no net.FileDialer (like there is net.FileListener) which would allow registering a fd or a raw syscall socket with the runtime network poller.

[b...@benburkert.com]

Yes, I have used the FileListener approach in the past. The annoying part was

duplicating the default socket options, which are platform specific.

I like the "Control ControlFunc" field on the Dialer approach, but it struck me

as odd to pass a syscall.RawConn callback into that, because of the Read &

Write methods. The syscall.Socket interface (should have been called

syscall.RawSocket as Socket is already taken) seems cleaner, but I guess the

TCPListener & UnixListener types already have SyscallConn methods.

But it looks like a common syscall.RawSocket interface would not work as both a

ListenSyscall parameter and ControlFunc parameter, because the first needs a

"Control(fd uintptr) error" method and the second needs

"Control(func (fd uintptr) error) error".

Would changing ControlFunc to "func(f *os.File) error" work? It mostly avoids

exposing the "fd uintptr" of the socket, and avoids the double callbacks.

[b...@benburkert.com]

> Should the user-defined control function be applied to derived sockets such as DNS transport for dialing?

I don't think the Control func should be called on a socket for DNS because the Dialer can set a Resolver, which has it's own Dial func field. This Dial func in turn can point to a Dialer that can sets a Control func for DNS sockets.

[Ian Lance Taylor]

On Tue, Oct 31, 2017 at 11:26 PM, <b...@benburkert.com> wrote:
>> Should the user-defined control function be applied to derived sockets
>> such as DNS transport for dialing?
>
> I don't think the Control func should be called on a socket for DNS because
> the Dialer can set a Resolver, which has it's own Dial func field. This Dial
> func in turn can point to a Dialer that can sets a Control func for DNS
> sockets.

I agree. I see no need for the Control function to apply to anything
other than the socket being created.

Ian

[Ian Lance Taylor]

One thing I like about the current proposal
(https://golang.org/issue/9661) is that it doesn't require any
additional packages. We already invented syscall.RawConn for a very
similar purpose, and this seems to me to be a minor extension of that
idea. Your idea is not bad, but it does seem to require another
package to create a syscall.Socket, and that seems like a complexity
we don't need.

Ian

[mikioh...@gmail.com]

Sorry for being late, a few random comments.

- I'm fine with having a user-defined function for controlling socket options on a larval, I mean, not attached to protocol control block inside the kernel, socket. My concern is the ambiguity of the larval socket. If we use the existing syscall.RawConn interface, the user-defined function have to do a blind take off because syscall.RawConn doesn't provide socket facility information, as in, address family/communication domain, communication semantics and protocol. I think it's better not to use syscall.RawConn for this purpose.

- If we need to have ListenControl, we also need ListenPacketControl. I personally prefer to add a new control knob like "type Announcer struct" that has both Listen and ListenPacket methods.

[Ian Lance Taylor]

On Fri, Nov 10, 2017 at 10:05 AM, <mikioh...@gmail.com> wrote:
>
> - I'm fine with having a user-defined function for controlling socket
> options on a larval, I mean, not attached to protocol control block inside
> the kernel, socket. My concern is the ambiguity of the larval socket. If we
> use the existing syscall.RawConn interface, the user-defined function have
> to do a blind take off because syscall.RawConn doesn't provide socket
> facility information, as in, address family/communication domain,
> communication semantics and protocol. I think it's better not to use
> syscall.RawConn for this purpose.

I don't understand why the blindness matters. The user is completely
in control of both how the socket is created and of the RawConn
interface to use. Why do we need to explicitly pass information like
address family to the user supplied function, when the user is
determining that information anyhow with the way that they call Dial
and Listen?

> - If we need to have ListenControl, we also need ListenPacketControl. I
> personally prefer to add a new control knob like "type Announcer struct"
> that has both Listen and ListenPacket methods.

That is probably OK with me. What would the struct fields be?

Ian

[mikioh...@gmail.com]

I don't understand why the blindness matters.  The user is completely
in control of both how the socket is created and of the RawConn
interface to use.  Why do we need to explicitly pass information like
address family to the user supplied function, when the user is
determining that information anyhow with the way that they call Dial
and Listen?

When using "tcp", Dial and Dialer.Dial make multiple candidate sockets, then pick only one inflight socket on the assumption that candidate sockets never interfere each other. If options set by user-defined functions make some interference in subsequent processes such as bind, Dial and Dialer.Dial may return weird errors. We already have https://github.com/golang/go/issues/18183, which is a side effect on providing dual stack sockets, and I personally don't want to make Dial and Dialer.Dial users confuse more.

 

That is probably OK with me.  What would the struct fields be?

A user-defined control function as a first step. Later, control knobs for a bit complicated, not single-shot, options such as TFO, MPTCP, UDP transport options.

[Ian Lance Taylor]

On Fri, Nov 10, 2017 at 11:35 AM, <mikioh...@gmail.com> wrote:
>
>> I don't understand why the blindness matters. The user is completely
>> in control of both how the socket is created and of the RawConn
>> interface to use. Why do we need to explicitly pass information like
>> address family to the user supplied function, when the user is
>> determining that information anyhow with the way that they call Dial
>> and Listen?
>
>
> When using "tcp", Dial and Dialer.Dial make multiple candidate sockets, then
> pick only one inflight socket on the assumption that candidate sockets never
> interfere each other. If options set by user-defined functions make some
> interference in subsequent processes such as bind, Dial and Dialer.Dial may
> return weird errors. We already have
> https://github.com/golang/go/issues/18183, which is a side effect on
> providing dual stack sockets, and I personally don't want to make Dial and
> Dialer.Dial users confuse more.

I am honestly not worried about confusing users who choose to use the
RawConn interface.

>> That is probably OK with me. What would the struct fields be?
>
>
> A user-defined control function as a first step. Later, control knobs for a
> bit complicated, not single-shot, options such as TFO, MPTCP, UDP transport
> options.

Absolutely the last thing I personally want to see in net is a struct
with fields for options like that. That just extends the net API
without bound.

Ian

[Audrius Butkevicius]

Sincere apologies for bumping the github issue. I've assumed I subscribed to the thread and was waiting for emails, but apparently not.

With what has been discussed so far, I agree with Ian's position that people using touching syscall.RawConn are in undefined territory.

Regarding Announcer, I think that would be a nice addition in general, as then we are not polluting package scope with extra functions, like the CL does now, and makes these type of changes easier. 
However as Ian said, it's not immediately obvious to me what other things would live on this struct.

There are also a few other internal things which might be worth doing, such as moving the ever growing list of parameters passed down multiple levels of functions into some sort of struct so that adding new things to be passed down would be easier.

Yet I suspect this should be a separate CL.

 

I am ready to deliver whatever solution is necessary, just let me know what you guys agree on, so I can get this done ASAP.

[mikioh...@gmail.com]

I am honestly not worried about confusing users who choose to use the
RawConn interface.

As far as I understand, RawConn interface is just for established, inflight sockets. I think that using RawConn interface is not appropriate for manipulating larval sockets. In addition, there are options that should be set during creation (e.g., UDP/UDP-lite, close-on-exec). So I'd rather see

// NewFile specifies a function that returns a new file containing

// a larval socket descriptor.

// It's caller's responsibility to make and configure an appropriate

// file for the arguments of network and address parameters, which

// specify network endpoint facilities.

type Dialer struct { NewFile func(network, address string) (*os.File, error) }

than

// Control optionally specifies a control function that is called after

// initialization of the socket, but before dialing.

// It's caller's responsibility to infer and manipulate the underlying

// network endpoint facilities of the raw connection without any hint.

type Dialer struct { Control func(syscall.RawConn) error }

[Russ Cox]

On Wed, Nov 15, 2017 at 8:38 PM, <mikioh...@gmail.com> wrote:

I am honestly not worried about confusing users who choose to use the
RawConn interface.

As far as I understand, RawConn interface is just for established, inflight sockets. I think that using RawConn interface is not appropriate for manipulating larval sockets.

I don't see why not.

 

In addition, there are options that should be set during creation (e.g., UDP/UDP-lite, close-on-exec). So I'd rather see

We always set close-on-exec; that's not something others need to or should mess with.

What do you mean by UDP/UDP-lite? How does that affect the socket syscall that creates the fd?

Russ

[mikioh...@gmail.com]

What do you mean by UDP/UDP-lite? How does that affect the socket syscall that creates the fd?

Using UDP-lite we need to specify the protocol number for UDP-lite instead of UDP like the following:

syscall.Socket(syscall.AF_{INET,INET6}, syscall.SOCK_DGRAM, 136)

[mikioh...@gmail.com]

As far as I understand, RawConn interface is just for established, inflight sockets. I think that using RawConn interface is not appropriate for manipulating larval sockets.

I don't see why not.

The syscall.RawConn interface never provides any hint for the underlying facilities. With a net.Conn/PacketConn containing an inflight socket, we can use  LocalAddr/RemoteAddr methods for getting the hint, but with a larval socket we cannot. I'm also fine if the proposal changes the signature from

type Dialer struct { Control func(c syscall.RawConn) error }

to

type Dialer struct { Control func(network, address string, c syscall.RawConn) error }

[Audrius Butkevicius]

Control func(network, address string, c syscall.RawConn) error

Is everyone happy with that, and ListenControl? 

[mikioh...@gmail.com]

Is everyone happy with that, and ListenControl? 

I still don't see any good reason to hesitate to use the existing signatures; func(string, string) (Listener, error) and func(string, string) (PacketConn, error), also I don't think that function values will harm us. Aside from that point, Im' fine that the revised cl72810 catches the go1.10 release train.

[Audrius Butkevicius]

I am not sure what you mean by using existing signatures, given there is no way to pass anything new apart to the existing signatures apart from adding variadic arguments?

[mikioh...@gmail.com]

On Monday, November 20, 2017 at 6:25:31 PM UTC+9, Audrius Butkevicius wrote:

I am not sure what you mean by using existing signatures, given there is no way to pass anything new apart to the existing signatures apart from adding variadic arguments?

I mean, https://golang.org/ref/spec#Method_expressions. For example,

type T struct { F func(...) error }

func (t *T) Listen(string, string) (Listener, error)

then you can pass (&T{}).Listen and net.Listen without any special care.

[Audrius Butkevicius]

Sorry, but I am failing to connect how this relates to the existing CL, as that still does not add an API to be able to set socket options.

If we add Announcer type in net, yes, that would work, but I think we've settled on ListenControl, with ControlFunc signature to be changed to

func(net, addr string, conn syscall.RawConn) error

Instead of

func(conn syscall.RawConn) error

Or am I misunderstanding something?

[mikioh...@gmail.com]

On Monday, November 20, 2017 at 8:45:49 PM UTC+9, Audrius Butkevicius wrote:

Sorry, but I am failing to connect how this relates to the existing CL, as that still does not add an API to be able to set socket options.

If we add Announcer type in net, yes, that would work, but I think we've settled on ListenControl, 

Sorry, but I don't think so.

iant> Absolutely the last thing I personally want to see in net is a struct with fields for options like that. That just extends the net API without bound. 

you> However as Ian said, it's not immediately obvious to me what other things would live on this struct.

[Audrius Butkevicius]

So I've updated the CL with my interpretation of what was discussed here.

I might be completely off the target, given there was a bit of back and forth happening here, and there wasn't a conclusion on what exactly needs to happen.

Let me know if anything needs to be changed, as I'd definately like this to land in 1.10

[Audrius Butkevicius]

So I've updated the CL with my interpretation (or misinterpretation) of what was discussed here. What happens now?

[Brad Fitzpatrick]

Looks like Russ replied on https://go-review.googlesource.com/c/go/+/72810 and marked it for Go 1.11.

On Mon, Nov 27, 2017 at 5:22 AM, Audrius Butkevicius <audrius.b...@gmail.com> wrote:

So I've updated the CL with my interpretation (or misinterpretation) of what was discussed here. What happens now?

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.