The syscall package will be frozen: A proposal

[Rob Pike]

Details at https://docs.google.com/a/golang.org/document/d/1QXzI9I1pOfZPujQzxhyRy6EeHYTQitKKjHfpq0zpxZs

-rob

[Jan Mercl]

Seems okay here.

(phone)

-j

[roger peppe]

will the standard library itself depend on go.sys?

On 17 July 2014 18:14, Rob Pike <r...@golang.org> wrote:
> Details at https://docs.google.com/a/golang.org/document/d/1QXzI9I1pOfZPujQzxhyRy6EeHYTQitKKjHfpq0zpxZs
>
> -rob
>

> --
> You received this message because you are subscribed to the Google Groups "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Brad Fitzpatrick]

No.

It says that it in the doc too.

[minux]

On Thu, Jul 17, 2014 at 2:00 PM, roger peppe <rogp...@gmail.com> wrote:

will the standard library itself depend on go.sys?

No. the standard library will use internal/syscall if the frozen syscall package

is not enough.

[Brendan Tracey]

I’m under the impression that some of the packages in go.* will migrate to the standard library when they mature. Presumably when those migrate they would change from using go.sys to internal/syscall?

[roger peppe]

ha, so it does. sorry for the noise.

[Brad Fitzpatrick]

Yes, if that ever happens. There is maybe only one I can think of that might, though.

[brainman]

SGTM

Thank you, Rob. Pretty much everything is covered. I have couple of questions.

Will Go code in go.sys/windows package live in ..._windows.go (or +build windows) files? We should do that for "go test go.sys/..." command to work properly on any platform. But then godoc will still need GOOS=windows if I am running it on linux.

Current syscall cannot be changed due to Go1 guarantee. Do you have a proposal on how to manage change in go.sys? Will be on "as needed" basis?

How we will decide what goes in? There are a few packages on the Net that provide interface to various Windows facilities (GUI, COM, Windows services and others). Are these fair go for inclusion? Maybe as a go.sys/windows/gui? I am not clear about what is "acceptable" for go.sys inclusion.

Alex

[minux]

I have one question:

even between different flavors of unixes, the syscall interfaces are still different.

for example, FreeBSD has capsicum, and solaris has a lot of more facilities than other unixes.

what about we take the time to make go.sys/unix only cover parts that are truly portable

across unixes and a separate package for each Unix that contains their own special interfaces?

i.e.

go.sys/unix   basic unix facilities

go.sys/linux  linux specific syscalls/structs

go.sys/freebsd freebsd specific ones

....

we can define that every OS specific directory are always auto-generated that will have native

look and feel (and will always follow their own naming convention).

whereas the unix package imports them all, and export one consistent interface (notably,

Atim and Atimespec field in Stat_t), the one could probably also can be automatically

generated. go.sys/unix should have tests for each exported functions and our builders

could verify it works on every unix system.

I imagine that in the future, we can have a go.sys/posix package that is portable across all

supported (mostly) POSIX compliant systems.

I understand that this is a lot more work, but it seems better for the users.

[Russ Cox]

It is an explicit goal not to be too fine-grained. Fine-grained distinctions end up wrong after minor changes to the things being distinguished. What's linux-specific today may be in freebsd and netbsd and openbsd tomorrow.

The basic unix, windows, plan9 separations make sense because there is close to 0 overlap, and what overlap there is is already exposed in portable ways by package os. The same argument (it's in package os) applies to go.sys/posix.

Don't overthink this.

Russ

[Rob Pike]

Alex,

Code in the go.sys packages will be conditionally compiled, just as
they are now. They must be, for correctness.

The go.sys packages will be maintained essentially as the syscall
package has been up to now: if it seems reasonable, it goes in. As I
said in the proposal, they may be more inclusive. You can call that
"as needed".

As for your final questions, I'm not sure, but those are problems for
another day. I can imagine a number of ways to go. Let's not worry
about it now.

-rob

[minux]

On Thu, Jul 17, 2014 at 9:01 PM, brainman <alex.b...@gmail.com> wrote:

How we will decide what goes in? There are a few packages on the Net that provide interface to various Windows facilities (GUI, COM, Windows services and others). Are these fair go for inclusion? Maybe as a go.sys/windows/gui? I am not clear about what is "acceptable" for go.sys inclusion.

Could we automate the generation of go.sys/windows from various header files?

If we can, then I suggest we do it once and for all by generating all API/structs that is in the SDK,

for all DLLs that are provided by the system.

The windows package is separate, so we don't need to worry about having a huge number of symbols

will slow down compilation, bloat the syscall package or anything like that.

If the DLL is bundled with the system, then it makes sense to go into go.sys/windows, this is my

suggested policy. (as long as the procedure is fully automated that doesn't require human intervention.)

This also saves us time: we don't need to decide whether this needs to go in or not.

Generally, I think we can adopt this to go.sys packages for all other systems: as long as it qualify

as syscall (for some system-depend definition of "syscall"), and we can automate generation of the

Go code, it could go in. For solaris and windows, this effectively means that all api provided by

system libraries. (We can use gccgo -fdump-go-spec to generate go definition from C definitions,

or go tool cgo -godefs).

For Unix, I think we can parse the kernel headers/syscall.master for the prototype (and we already

have script for this), and manpages for the docs (in fact, we can even extract docs for each individual

struct field and constant from the manpages). This seems like a fun project to do, btw.

[brainman]

On Friday, 18 July 2014 15:28:56 UTC+10, Rob Pike wrote:

Thank you for your comments.

Alex

[brainman]

> On Friday, 18 July 2014 15:46:46 UTC+10, minux wrote:

> Could we automate the generation of go.sys/windows from various header files?

It is possible, but I don't see the need. It is not difficult for me to translate odd API when I need them the way we do now.

> If we can, then I suggest we do it once and for all by generating all API/structs that is in the SDK,

> for all DLLs that are provided by the system.

I am not sure we can. But, like I said before, I don't see the need.

> The windows package is separate, so we don't need to worry about having a huge number of symbols

> will slow down compilation, bloat the syscall package or anything like that.

I don't see who will use windows package.

> If the DLL is bundled with the system, then it makes sense to go into go.sys/windows, this is my

> suggested policy. (as long as the procedure is fully automated that doesn't require human intervention.)

What does "DLL is bundled with the system" means? How're you going to decide if "DLL is bundled with the system"?

> This seems like a fun project to do, btw.

I don't dispute that. I tried something similar recently - IDispatch::GetTypeInfo.

Alex

[Aram Hăvărneanu]

This looks very good, thank you.

I'm wondering if we can't prune the existing syscall package a
little. At first glance, he Go1 compatibility guarantee would appear
to prevent us from removing anything, but there are some system
calls which can't be called by client code at all. All code that
calls them is wrong, with bugs lurking. I'm thinking of system calls
like clone(2).

I have seen code that mindlessly calls clone(2) in the wild.

Another thing I am wondering is whether I can remove pretty much
the entiretly of the syscall package on Solaris. Again, the Go1
compatibility guarantee would appear to prevent us, but Solaris is
not in go/api at all. This is by design, not because I forgot to
add it there; I wasn't happy with the package in the middle of the
1.3 cycle, and I wanted to change it. Unfortunately, neither myself
nor anyone else had a better idea, but now this proposal materialized.

Another thing I am wondering is whether to create a gofix module
(perhaps off by default) that changes client to use go.sys instead
of syscall. Technically it's not needed, since we don't break any
code, however, it's a nice addition and a testament that the Go
team cares about the lifetime of client code.

--
Aram Hăvărneanu

[Chris Hines]

On Friday, July 18, 2014 2:59:40 AM UTC-4, brainman wrote:

> On Friday, 18 July 2014 15:46:46 UTC+10, minux wrote:

> Could we automate the generation of go.sys/windows from various header files?

It is possible, but I don't see the need. It is not difficult for me to translate odd API when I need them the way we do now.

 

> If we can, then I suggest we do it once and for all by generating all API/structs that is in the SDK,

> for all DLLs that are provided by the system.

I am not sure we can. But, like I said before, I don't see the need.

> The windows package is separate, so we don't need to worry about having a huge number of symbols

> will slow down compilation, bloat the syscall package or anything like that.

I don't see who will use windows package.

I hope I haven't misunderstood the scope of this comment, but I would use it.

Of course, that depends on what APIs it covers. There are a handful of Windows APIs that I have needed that the current syscall doesn't cover. Specifically the APIs related to Job Objects and file sharing/locking. I may need others in the future.

On the other hand, it wasn't that hard for me to generate my own wrappers for the APIs that I needed--once I figured it out. Figuring out how to do it was the challenging part. It is sparsely documented and required using a Perl script. In addition I had to modify the Perl script in order to get the package scoping correct since my wrappers were not in syscall proper, but used things like syscall.Handle. I understand the perl script has been ported to Go, but last I checked it was still focused only on generating code for syscall.

So, I would prefer to see:

1. Package go.sys/windows has everything I need, or

2. There is documented and reasonably easy to use tooling for me to generate my own wrapper packages on demand, ideally something I could hook into the proposed go generate framework. In addition, the generated wrappers should be able to inter-operate with go.sys/windows code when necessary.

[Nick Craig-Wood]

On 17/07/14 18:14, Rob Pike wrote:
> Details at https://docs.google.com/a/golang.org/document/d/1QXzI9I1pOfZPujQzxhyRy6EeHYTQitKKjHfpq0zpxZs

[from the doc]
> Inside go.sys, there will be three packages, independent of syscall, called plan9, unix, and windows, and the current syscall package's contents will be broken apart as appropriate and installed in those packages.

It seems to me that not having a package per unix architecture
potentially leads back to problem 4

> The single package, called syscall, is different for every system, but godoc only shows the variant for its own native environment

Namely that there are things present in the unix package which work for
linux, say but missing for darwin.

A single unix package would be easier to use, but only if it was
specified that anything in it had to be working for all unix like
architectures. If not then my opinion is that it should go in a linux
or darwin package.

--
Nick Craig-Wood <ni...@craig-wood.com> -- http://www.craig-wood.com/nick

[Rob Pike]

The majority of the syscall interface, dynamically speaking, is
identical at the source level across the systems. The differences are
easily handled by build tags, as they are today.

-rob

[Michael MacInnis]

Just to be clear, does this mean that changes to SysProcAttr (specifically, the Foreground, Joinpgrp and Sigdfl members I was hoping could be added) will have to wait until after Go 1.4?

[Andrew Gerrand]

No, you can add them to the new go.sys packages as soon as they are available, which should be in the next couple of months. 

Andrew

On Sunday, 20 July 2014, Michael MacInnis <michael.p...@gmail.com> wrote:

Just to be clear, does this mean that changes to SysProcAttr (specifically, the Foreground, Joinpgrp and Sigdfl members I was hoping could be added) will have to wait until after Go 1.4?

[brainman]

On Saturday, 19 July 2014 00:43:29 UTC+10, Chris Hines wrote:

> I understand the perl script has been ported to Go, but last I checked it was still focused only on generating code for syscall.

It is my mistake. I create an issue to track this https://code.google.com/p/go/issues/detail?id=8398. And I have a fix for it too https://codereview.appspot.com/114140043 Would you like to review CL?

Alex

[Chris Hines]

Thanks, Alex, my comments are on the code review.

[Mikio Hara]

lgtm, agree that we need some guideline. the minimum requirement for
the surface might be exposing syscall.Syscall{,6,9}. hope that we have
a chance to move syscall.Socket stuff into the net or
appropriate/intermediate package in go2 (golang.org/issue/5616).

ps: fwiw, linux/386 doesn't expose syscall.Syscall{,6,9} for socket in
go1.3. it might be a problem for someone who wants to dance with
runtime using own syscall stuff, but we're not sure the platform lasts
beyond CY2038, so i think it's fine. (and i just dropped the support
for linux/386 from my experiments :)