syscall.SetNonblock stopped working in Go 1.9.3

[Dave Keck]

Hey all, this program exits (as expected) when run with macOS 10.12.6 + Go 1.8.6, but it deadlocks when run with Go 1.9.3:

    https://play.golang.org/p/Dw_ND9gkgPm

The same behavior is observed when using unix.SetNonblock instead of syscall.SetNonblock.

It appears that the SetNonblock() functions don't have an effect in Go 1.9.3. Is this a known issue?

Thanks!

David

[Steven Hartland]

I checked this on FreeBSD and it still works, so seems like it could be a macOS specific issue.

You should raise a bug here:
https://github.com/golang/go/issues

It would be good to try and identify the go version which it first broke in too, as that will be helpful in identifying the change which cause the regression.

    Regards
    Steve

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Max]

I tried on Linux/amd64 (debian testing, kernel 4.14.0-2-amd64) and it's broken too:

* go 1.9.3 + syscall.SetNonblock : hangs in syscall.Read()
* go 1.9.3 + unix.SetNonblock    : hangs in syscall.Read()
* go 1.9.2 + syscall.SetNonblock : hangs in syscall.Read()
* go 1.9.2 + unix.SetNonblock    : hangs in syscall.Read()

the relevant strace(1) fragment is:

-----------------------------------
pipe2([3, 4], O_CLOEXEC)                = 0
epoll_create1(EPOLL_CLOEXEC)            = 5
epoll_ctl(5, EPOLL_CTL_ADD, 3, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2231488256, u64=140400417427200}}) = 0
fcntl(3, F_GETFL)                       = 0 (flags O_RDONLY)
fcntl(3, F_SETFL, O_RDONLY|O_NONBLOCK)  = 0
epoll_ctl(5, EPOLL_CTL_ADD, 4, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2231488064, u64=140400417427008}}) = 0
fcntl(4, F_GETFL)                       = 0x1 (flags O_WRONLY)
fcntl(4, F_SETFL, O_WRONLY|O_NONBLOCK)  = 0
fcntl(3, F_GETFL)                       = 0x800 (flags O_RDONLY|O_NONBLOCK)
fcntl(3, F_SETFL, O_RDONLY)             = 0
fcntl(3, F_GETFL)                       = 0 (flags O_RDONLY)
fcntl(3, F_SETFL, O_RDONLY|O_NONBLOCK)  = 0
write(1, "Read START\n", 11Read START
)            = 11
fcntl(3, F_GETFL)                       = 0x800 (flags O_RDONLY|O_NONBLOCK)
fcntl(3, F_SETFL, O_RDONLY)             = 0
read(3,
<hangs here>
-----------------------------------

I hope this helps.

David, did you already report the bug on https://github.com/golang/go/issues ?
I can add the details above on the bug report too.

Regards,

Max

[Max]

Looking at the Linux stack trace, the culprit seems the next-to-last line:
-------------

fcntl(3, F_SETFL, O_RDONLY)             = 0
read(3, <hangs here>
-------------

i.e. at least on Linux the O_NONBLOCK bit gets cleared immediately before the read()

[Ian Lance Taylor]

As of 1.9 os.Pipe returns non-blocking descriptors by default. In
order to be consistent with past releases, when you call the Fd
method, the descriptor is explicitly set back into blocking mode,
since that is how Fd behaved in the past. In your code you are
calling
syscall.Read(int(pr.Fd()), make([]byte, 1, 1))
so the effect of calling pr.Fd() is to undo the earlier call to
syscall.SetNonblock.

This is an unfortunate consequence of reasonable decisions.
Fortunately the fix for your code is simple: write `fd := pr.Fd()` and
then use that, rather than calling the Fd method several times. Or,
perhaps, just trust that pipe's won't be in blocking mode and let the
Go runtime handle them efficiently, rather than trying to do your own
nonblocking I/O.

Ian

[Max]

Thanks for the explanation and the solution :)

Where is this behaviour documented?

The current documentation https://golang.org/pkg/os does not indicate
blocking vs non blocking behaviour of *os.File and of functions that return
*os.File, as for example os.Pipe()

Also, os.File.Fd() having side effects seems very unexpected and surprising at least to me.

Best regards,

Max

[Ian Lance Taylor]

On Thu, Jan 25, 2018 at 7:04 AM, Max <massimilia...@gmail.com> wrote:
>
> Thanks for the explanation and the solution :)
>
> Where is this behaviour documented?
>
> The current documentation https://golang.org/pkg/os does not indicate
> blocking vs non blocking behaviour of *os.File and of functions that return
> *os.File, as for example os.Pipe()
>
> Also, os.File.Fd() having side effects seems very unexpected and surprising
> at least to me.

I found it very difficult to write fully correct documentation for
this. Explaining it overwhelmed the important points with detail that
is irrelevant to 99.9% of users. All that is there at the moment is
the docs on the Fd method that say "On Unix systems this will cause
the SetDeadline methods to stop working." I'm certainly open to
suggestions but anything added has to be very short and very clear.
Adding a sentence like "the file descriptor is returned in blocking
mode" leads you into a rathole of trying to explain which files are
blocking and which are not, and what that even means in a language
like Go.

More broadly, the general guideline is always going to be to not try
to flip back and forth between the os package and the syscall package.
Pick one or the other. When you need to switch, use Fd or NewFIle,
once.

Ian

[Peter Mogensen]

On 2018-01-25 16:18, Ian Lance Taylor wrote:
> On Thu, Jan 25, 2018 at 7:04 AM, Max <massimilia...@gmail.com> wrote:
>> Also, os.File.Fd() having side effects seems very unexpected and surprising
>> at least to me.

.File() on a network conn have had that effect.

> I found it very difficult to write fully correct documentation for
> this. Explaining it overwhelmed the important points with detail that
> is irrelevant to 99.9% of users. All that is there at the moment is
> the docs on the Fd method that say "On Unix systems this will cause
> the SetDeadline methods to stop working."

Better docs would be great. I recall spending a lot of time figuring
this out.
I had the need to do .File().Fd() on a network socket.
- not to do any operations on it, but just to keep a dup(2) around
until the original FD was closed.

So the solution ended up being doing SetNonblock() ... but it was not
easy to see from the docs.

/Peter

[Dave Keck]

Thanks all. I opened this issue:

    https://github.com/golang/go/issues/23556

(The reason I'm using SetNonblock is that before Go 1.9, it wasn't safe to call file.Close() if another goroutine was reading/writing to the pipe. But now that it's safe with Go 1.9, I can re-organize my code and remove the SetNonblock calls.)

David

On Thu, Jan 25, 2018 at 9:34 AM, Peter Mogensen <a...@one.com> wrote:

On 2018-01-25 16:18, Ian Lance Taylor wrote:

> On Thu, Jan 25, 2018 at 7:04 AM, Max <massimiliano.ghilardi@gmail.com> wrote:

>> Also, os.File.Fd() having side effects seems very unexpected and surprising
>> at least to me.

.File() on a network conn have had that effect.

> I found it very difficult to write fully correct documentation for
> this.  Explaining it overwhelmed the important points with detail that
> is irrelevant to 99.9% of users.  All that is there at the moment is
> the docs on the Fd method that say "On Unix systems this will cause
> the SetDeadline methods to stop working."

Better docs would be great. I recall spending a lot of time figuring
this out.
I had the need to do .File().Fd() on a network socket.
 - not to do any operations on it, but just to keep a dup(2) around
until the original FD was closed.

So the solution ended up being doing SetNonblock() ... but it was not
easy to see from the docs.

/Peter

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.