Moving Go to a newer NetBSD ABI in the 1.13 dev cycle

[Benny Siegert]

Hi and apologies for the cross-post,

Go binaries have always been somewhat special citizens on NetBSD (not
unlike on other OSes, I might add). Currently, IIUC, they all pretend
to be built for NetBSD 5.99.51, since that's what I was using when we
were getting the support going, so this was the oldest version known
to work.

However, Go 1.9 is really the last version that was working on NetBSD
6, 1.10 and newer effectively need NetBSD 7 or later. This is because
of some kernel bugfixes. NetBSD 7 is the oldest supported version at
this point, though support is rapidly waning as (a) NetBSD 9 is coming
nearer (which will mean end of support for -7) and (b) NetBSD 7 has no
Spectre/Meltdown mitigations.

Being soi-disant compatible with 5.99.51 has a cost, namely the need
for COMPAT_60 to be enabled in the kernel.

To keep up with supported versions, I would like to move Go to use
syscall number tables for NetBSD 7 (or perhaps even 8?) and move the
ELF identification to version 7.0 or 8.0. Trying to get this done in
time for Go 1.13.

Any comments, objections, etc. from Go or NetBSD folks?

--
Benny

[Aram Hăvărneanu]

SGTM.

--
Aram Hăvărneanu

[David Chase]

Is anyone still running the lower-numbered NetBSDs? 6 is not maintained, 7.0 was released September, 2015, 8.0 July 2018.

I lean in favor of the change, 1.13 occurs about a year after 8.0's release, what has the adoption been like?

On Thu, Mar 7, 2019, 7:25 AM Aram Hăvărneanu <ara...@mgk.ro> wrote:

SGTM.

--
Aram Hăvărneanu

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Austin Clements]

Sounds good to me.

[Kamil Rytarowski]

We should switch to libc calls and reduce users of syscall(2)/__syscall(2).

[Greg Troxel]

Benny Siegert <bsie...@gmail.com> writes:

> To keep up with supported versions, I would like to move Go to use
> syscall number tables for NetBSD 7 (or perhaps even 8?) and move the
> ELF identification to version 7.0 or 8.0. Trying to get this done in
> time for Go 1.13.
>
> Any comments, objections, etc. from Go or NetBSD folks?

I think you should try to keep go working on NetBSD 7. While the branch
is, someday, going to to stop receiving fixes, it seems a bit much to
have a language necessary for significant things stop working unless
there are really compelling reasons.

As I understand you, the only reason to pick 8 instead of 7 is to avoid
having to have some kind of COMPAT_7 option in the kernel. Surely 8 and
almost certainly 9 are going to have that, so that people can upgrade.
So I don't see the gain other than perhaps reducing the time to the next
bump.

I don't understand your point about 7 and Spectre/Meltdown. That's of
course one of many reasons for people to upgrade, but I don't see that
it's a reason to break go on 7, harming people that are making choices
to stay on 7 due to considerations we don't know about. As I see it,
it's ok to desupport go on a system when the pain of keeping it working
outweights the pain of the people that lose it, adjusted by some sort of
metric about whether they should instead update. People running 6
should already have updated, but 7 is today stable and supported.

Also, I don't understand go innards, but it would be nice to not need
this kind of pinning.

Is it possible to set a target ABI that is different on different
systems, building go on 7 for 7, and on 8 for 8? That must be hard, or
you wouldn't be asking what you are asking...

g...@NetBSD.org (resident old system and portability crank)

[ma...@netbsd.org]

sounds good. it's not a big change. I don't think a "wrong" OS version
is rejected.
the changes will probably be:
netbsd-8 has some routing socket stuff(?)
changing the OS version in the binary will change the x86 float control
word.

[Guy Brandwine]

First, thanks for the heads up.

My personal concern is the lack of a policy or just good ground rules for removing the support for specific os's.

Such a policy would allow developers to prepare, whether their in the dev group or not, and not be surprised when the certain is out.

Moreover, when running services on the cloud, the chance that you will use end of life os is low, and upgrade is relatively easy.

As someone who's doing business with global enterprises (a spot where go is still not catching massively) with thousands of servers some are antique, I can tell you that each time  a support for an Os drops (centos 5, Windows XP), requesting the customer to survey and replace/upgrade in a short period of time is a Hussle (alternatively continue using older go versions is a a lame practice )

I think we should set up a criteria based on:

1. End of life date

2. Market share.

3. The reason for removing the support.

If we can create such a table, developers can raise their requests/thoughts early, and prep ahead.

Sorry for going a little bit off topic.

[Ian Lance Taylor]

Go does work that way on Darwin, Solaris, AIX, and Windows. It has an
efficiency cost--Go programs run somewhat slower--and it means that
someone has to write a bunch of near-boilerplate code (see, e.g.,
runtime/sys_darwin.go). Is there a reason to go this route for
NetBSD? Is NetBSD trying to avoid statically linked executables?

Ian

[Martin Husemann]

This is unrelated to statically linked executables (we do provide
a static libc).

The point is more the very fragile syscall API, especially when you try
to write portable code (that is: same syscall, but portable over a variety
of architectures).

We have for example this hack in our __syscal test in the NetBSD test suite:

#if !defined(_LP64) && BYTE_ORDER == _BIG_ENDIAN
#define __SYSCALL_TO_UINTPTR_T(V) ((uintptr_t)((V)>>32))
#else
#define __SYSCALL_TO_UINTPTR_T(V) ((uintptr_t)(V))
#endif

and then do this to invoke mmap via __syscall:

p = (const char *)__SYSCALL_TO_UINTPTR_T(__syscall(SYS_mmap,
0, sizeof(secrect_data), PROT_READ, MAP_PRIVATE, fd,
/* pad*/ 0, (off_t)0));

So overall this is considered a hack (or at least very ugly), and using
the (static) libc stub is a lot easier to get right and portable.
Of course this mostly applies to C usage, and if you got the go glue
code right for an architecture once, you are done ;-)

Martin

[ma...@netbsd.org]

Portability: Go right now doesn't take advantage of this portability, at
least from an outsider perspective it looks like it might have been
hastily done to solve an urgent issue for the OSes that need it.

Compatibility: since Go doesn't read C headers, and NetBSD does updates
the functions using macro renaming, it won't use the newer version, so
no benefits here.

It adds some work for a regression in performance.

NetBSD is decently backwards compatibility on syscalls so it's not
needed, although mistakes have happened in the past.

[Kamil Rytarowski]

This is right. It's hard to write portable code to multiple ABIs using
syscall(2)/__syscall(2) hack. It's also not possible to express all
types of syscalls (returning multiple values).

In my opinion, slight cost of performance is superior to maintenance
burden of indirect syscall API in C.

We had to switch sanitizers to libc calls, because there were too many
issues with syscall(2)/__syscall(2). There were also discussions about
dropping this API from newer kernels (I support it), and we need to
switch current users away.

[Martin Husemann]

On Fri, Mar 08, 2019 at 10:22:07AM +0100, Kamil Rytarowski wrote:
> In my opinion, slight cost of performance is superior to maintenance
> burden of indirect syscall API in C.

I am not sure where this slight performance hit would come from, especially in
the context of statically linked binaries.

Martin

[ma...@netbsd.org]

The proposed options for this scenario (judging by the Darwin code) are:
Go code -> Go ABI to C ABI translation -> syscall stub in libc -> syscall

vs.
Go code -> Go ABI syscall stub -> syscall

(wild option for portability but not compatibility*, using the
Syscall{,6,9} functions with SYS_syscall more, which is probably less
heavy)

* Which we won't gain anyway, see:
http://mail-index.netbsd.org/tech-pkg/2019/03/08/msg020917.html

[Martin Husemann]

On Fri, Mar 08, 2019 at 09:49:21AM +0000, ma...@netbsd.org wrote:
> The proposed options for this scenario (judging by the Darwin code) are:
> Go code -> Go ABI to C ABI translation -> syscall stub in libc -> syscall
>
> vs.
> Go code -> Go ABI syscall stub -> syscall

There is missing the system call specific stuff between the first two
links here, and the Go-ABI-to-system-call-abi translation in the second one,
so it is not clear at all to me which method would perform better.

All these layers/shims are pretty tiny, so I wouldn't be suprised if there
would be no measurable difference (but I'd love to see hard numbers).

> (wild option for portability but not compatibility*, using the
> Syscall{,6,9} functions with SYS_syscall more, which is probably less
> heavy)
>
> * Which we won't gain anyway, see:
> http://mail-index.netbsd.org/tech-pkg/2019/03/08/msg020917.html

Sorry, I fail to parse this (including your tech-pkg post).

Martin

[Aram Hăvărneanu]

> All these layers/shims are pretty tiny, so I wouldn't be suprised if there
> would be no measurable difference (but I'd love to see hard numbers).

The overhead doesn't come from the ABI translation, but it comes
from the Go runtime. If libc is not used, Go can do system calls
on Go stacks. If libc is used, Go can't do system calls on Go stack,
it needs to switch to a C-compatible stack. Before switching to a
C stack, the Go program must inform the runtime of this, then, the
runtime will take certain actions. Similarly, after return from
system call, Go will switch stacks and the runtime must be informed
again and it will take other actions.

It is this overhead in the runtime that make the Go-C bridge
relativelly slow, not the ABI translation itself.

This overhead is still pretty minor though. Its effects are usually
felt when calling high-performance user space C code, not when doing
system calls, where the execution of the system call will usually
dwarf this overhead.

Note that the Solaris port is 95% a POSIX port. There are few Solaris
specific parts. If the Go port to NetBSD switched to libc model we
could remove a lot of code, since the Solaris code should "just
work". Same for every other Unix port really.

--
Aram Hăvărneanu

[Greg Troxel]

David Chase <drc...@google.com> writes:

> Is anyone still running the lower-numbered NetBSDs? 6 is not maintained,
> 7.0 was released September, 2015, 8.0 July 2018.

A fair question.

I'm sure some people are still running 6.

Many people delay upgrading across major versions, and last July is
really pretty recent. This is a bit like people running LTS versions of
RHEL, where they only want base system security updates and avoid the
effort of major upgrades. This is particularly apt in production
situations with strained sysadmin resources, where people sometimes
upgrade every other branch.

So running 7 is pretty common, and it's not yet out of line even a
little bit. I myself have a 7 system, even though most are 8, and I
don't yet feel bad about that.

> I lean in favor of the change, 1.13 occurs about a year after 8.0's
> release, what has the adoption been like?

I think it's reasonable to desupport 6 as of about now. It's been out
of security maintenance for 8 months.

I would say that more people have upgraded to 8 than are still on 7. I
would even guess it's 75%. But it's nowhere near 99%.

A similar question might be how many people are running Ubuntu older
than 18.04 (all earlier verisons, including LTS)? I bet it's far from
negligible.

I have not seen an explanation of why desupporting 7 is useful for the
go world. From the NetBSD point of view, expecting people running 8, 9
(future, no date set) and -current to have COMPAT_70 in their kernels
(perhaps via modules) seems vastly better than breaking go on all 7
systems. The NetBSD 8 GENERIC for amd64, as an example, has compat
going back to 1.5. While some might consider that extreme, dropping
compat for two or three releases back seems unthinkable.

Greg

[Kamil Rytarowski]

On 08.03.2019 10:59, Martin Husemann wrote:
> On Fri, Mar 08, 2019 at 09:49:21AM +0000, ma...@netbsd.org wrote:
>> The proposed options for this scenario (judging by the Darwin code) are:
>> Go code -> Go ABI to C ABI translation -> syscall stub in libc -> syscall
>>
>> vs.
>> Go code -> Go ABI syscall stub -> syscall
>
> There is missing the system call specific stuff between the first two
> links here, and the Go-ABI-to-system-call-abi translation in the second one,
> so it is not clear at all to me which method would perform better.
>
> All these layers/shims are pretty tiny, so I wouldn't be suprised if there
> would be no measurable difference (but I'd love to see hard numbers).
>

The usage of syscall(2)/__syscall(2) might have tiny overhead on the
kernel side when compared to direct syscalls. In the end difference is
most likely negligible.

[Ian Lance Taylor]

For the Go 1.12 release we changed Darwin from using direct system
calls to going through Darwin's libSystem. According to
https://github.com/golang/go/issues/30497#issuecomment-468546122 the
slowdown was in the neighborhood of 20% on an I/O heavy benchmark.

Ian

[Rhialto]

On Fri 08 Mar 2019 at 13:58:02 +0100, Aram H?v?rneanu wrote:
> The overhead doesn't come from the ABI translation, but it comes
> from the Go runtime. If libc is not used, Go can do system calls
> on Go stacks. If libc is used, Go can't do system calls on Go stack,
> it needs to switch to a C-compatible stack. Before switching to a
> C stack, the Go program must inform the runtime of this, then, the
> runtime will take certain actions. Similarly, after return from
> system call, Go will switch stacks and the runtime must be informed
> again and it will take other actions.

Just out of curiosity, doesn't Go follow the ABI as established for the
CPU? I'm not sure where it is formally described, but
https://wiki.osdev.org/System_V_ABI seems a fairly general location.
Or maybe
https://software.intel.com/sites/default/files/article/402129/mpx-linux64-abi.pdf
for more a x86_64-only version. These documents describe function
calling conventions, including stack layout.

I would expect that if the established ABI is followed, then stacks are
compatible. Or am I too optimistic?

-Olaf.
--
___ Olaf 'Rhialto' Seibert -- "What good is a Ring of Power
\X/ rhialto/at/falu.nl -- if you're unable...to Speak." - Agent Elrond

[Aram Hăvărneanu]

> Just out of curiosity, doesn't Go follow the ABI as established for

> the CPU? [...] I would expect that if the established ABI is followed,

> then stacks are compatible. Or am I too optimistic?

No, Go currently uses a custom ABI with copying stacks that can
grow and shrink on demand: http://golang.org/s/contigstacks.
Previously it used split stacks (document is for GNU GCC/gold, but
similar enough): https://gcc.gnu.org/wiki/SplitStacks.

C code can't run on Go stacks because C code expects megabytes of
stacks. Go stacks start at kilobytes.

--
Aram Hăvărneanu