Help: Converting everything on the G stack to Go.

[Keith Randall]

For 1.4 we're working on converting a bunch of the runtime code from C to Go.  The goal is to have exact stack maps for every stack frame visible to the garbage collector or the stack copier.  This requirement means that everything on the G stack must be written in Go, as only the Go compiler knows how to generate stack maps.  The notable exceptions from this requirement are functions that run on the G stack but can't be seen by the collector/copier (for example, leaf nosplit functions), and assembly functions which have hand-coded stack maps (for example, the reflect.callXX functions.)

I've been working over the summer on a lot of the more complicated rewrites, for instance hash maps (CL 99380043), string operations (CL 93380044), interface conversions (CL 98510044), and so on.  We're nearing the finishing line where we have a lot of less complicated conversions still to do, and not much time left to do them.  That's where you come in.  I've made a list of the remaining work and I am asking for your help to complete it.  Go ahead and pick a chunk of whatever size you think you can handle, claim it on the spreadsheet, and dig in.

spreadsheet: https://docs.google.com/spreadsheets/d/1dQ8_hcNYYOKs2BszXUWvxnplsrTwNheVxgzr4y01ogw/edit?usp=sharing

Most of the conversions required are in .goc files.  Hopefully at the end of this process we will have only a single .goc file left, stubs.goc, which will contain only recursively nosplit functions. There are a few other conversions which are not in .goc files.  I've listed those separately.

Removing a C function from the G stack can be done in a few ways:

1. Convert the entire function to Go.  Make a .go file corresponding to the .goc file and move the implementation there.
2. Convert part of it, calling out to C on the M stack using runtime/stubs.go:onM to do the rest of the work in C.  There are examples of how to use onM in runtime/malloc.go and elsewhere.  Put the C code that runs on the M stack in a .c file.
3. Leave the function in C, but arrange to have it and all of its callees, recursively, labeled as nosplit.

Go definitions of runtime data structures are in the generated file runtime/zruntime_defs_GOOS_GOARCH.go.  You'll want to use these definitions in your Go code.

You do not need to convert any nosplit leaf functions.  For example, runtime/stubs.goc:acquirem.  Also, nosplit functions which call only other nosplit functions, and recursively, are also ok.  An example of this is runtime/stubs.goc:gocas, as it only calls cas which is itself a nosplit leaf function.  If you find functions that require no changes, just mark them as done on the spreadsheet.

This farming out of work is an experiment of a sort, so I'm happy to answer any questions you have.  Also, if you notice any functions you think need converting but are not covered by the spreadsheet, let me know.

-Keith

[Dave Cheney]

Thanks Keith.

I have a memory that there is a mode we can put the runtime into where it will log or panic if C code ends up on the Go stack. Is that true? Is such a thing possible ?

Dave

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Daniel Morsing]

Thanks Keith.

What's the plan for C "..." args in deferproc and newproc? Switching to the M stack with a pointer to the args might not work since the stack can move from underneath the C code.

Regards,

Daniel Morsing

[Dmitry Vyukov]

If deferproc is NOSPLIT and you call runtime.onM in it, then you are
on g0 without possibility of rescheduling/gcing/stackshrinking in
between. So you still have a valid args pointer into g stack.

[ericdr...@gmail.com]

Hi Keith:

I'm trying to write more and more code in Go and would like to help.  I'm concerned that I don't have the background for this task however.

I have a fairly good background in C and have been writing Go as much as possible over the last 12 to 15 months.

However, I have no idea what a G stack or a M stack is and it doesn't appear that there's a lot of material beyond the code to get one up to speed.  

Is there too much of a learning curve to take on something like this and contribute before it's done.

I understand this is somewhat subjective.  I'm just looking for some guidance.

Thanks 

Eric

[Dmitry Vyukov]

Is there anything special about calling Go code on g0 stack long-term?

[Dmitry Vyukov]

the table is missing functions from proc.c - newproc, ready
some of them are in miscellaneous, but that's also significant chunk of work

On Wed, Aug 20, 2014 at 11:01 AM, 'Keith Randall' via golang-dev
<golan...@googlegroups.com> wrote:

[Keith Randall]

> I'd like to take mprof.goc, i'm not able to edit the spreadsheet right now.
> 
> Rémy.

Rémy, I added you.

> I have a memory that there is a mode we can put the runtime into where it will log or panic if C > code ends up on the Go stack. Is that true? Is such a thing possible ?

No such mode currently exists.  I have a CL which does something like this, in the linker record (and eventually, forbid) when Go code calls C code.  There are a few exceptions that need to be hardcoded, like mcall and newstack.  If we get this all cleaned up for 1.4 we can enable it to prevent regression.

> Is there anything special about calling Go code on g0 stack long-term?

I have a vision of a strict separation, Go code only on the G stack and C code only on the M stack (and a few assembly routines, like memmove, allowed on both).  But that's harder and won't happen for 1.4.  In particular the Go malloc code runs on the M stack right now.

While there is Go code on the M stack, we do have to ensure that that Go code uses bounded stack.

> However, I have no idea what a G stack or a M stack is and it doesn't appear that there's a lot > of material beyond the code to get one up to speed.  

> 

> Is there too much of a learning curve to take on something like this and contribute before it's done.

> 

> I understand this is somewhat subjective.  I'm just looking for some guidance.

Unfortunately, this task has a steep learning curve for people who haven't worked on the runtime before.

[brainman]

I am trying to convert syscall_windows.goc. The file belongs to "syscall" package, not "runtime". I can move that file to "syscall", but it uses some internal runtime types, variables and functions. How do you propose I approach that? asm?

Thank you.

ALex

[minux]

On Aug 20, 2014 8:48 PM, "brainman" <alex.b...@gmail.com> wrote:
> I am trying to convert syscall_windows.goc. The file belongs to "syscall" package, not "runtime". I can move that file to "syscall", but it uses some internal runtime types, variables and functions. How do you propose I approach that? asm?

i think the only way to go is to move it to syscall, write in Go that calls assembly forwarding function (just jmp, don't use multiple files for each arch., use #ifdef GOARCH_arm and #define JMP B) to the runtime internal functions.

[brainman]

> ... use #ifdef GOARCH_arm and #define JMP B)

care to point me to an example file?

Alex

[minux]

:

On Aug 21, 2014 12:38 AM, "brainman" <alex.b...@gmail.com> wrote:
> > ... use #ifdef GOARCH_arm and #define JMP B)
> care to point me to an example file?

i just realized that we don't have windows/arm support atm, but anyway, here is one example:
http://tip.golang.org/src/pkg/os/signal/sig.s

it builds on all three supported architectures (not power64 though, because it uses BR, but don't worry about power64 now)

[brainman]

On Thursday, 21 August 2014 15:12:59 UTC+10, minux wrote:
> ... here is one example:
http://tip.golang.org/src/pkg/os/signal/sig.s

Thank you.

Alex

[sanjay.m]

Thanks for the example; I am having the same problem with rdebug.goc (it declares symbols for runtime/debug, and uses runtime internal functions).

To clarify, if the runtime-internal function is a Go function and the function take an argument, does the assembly stub still say "NOSPLIT,$0"? 

I ask because the NOSPLIT docs seem to say that the NOSPLIT-ted function's frame needs space for all called functions, but I feel like that isn't true if a called function does the stack split check in its own preamble.

Thanks,

Sanjay

[Dmitry Vyukov]

The better options is to keep that file in runtime, and use the asm
thunks to call unexported runtime functions from syscall package.
I have not looked at that particular case, but I think that
syscall<->runtime interface is smaller and better defined and does not
use runtime-internal types.

[Dmitry Vyukov]

see an example in https://codereview.appspot.com/127460044/ in reflect asm files

[Dmitry Vyukov]

Alex, example of such approach is in
https://codereview.appspot.com/127460044/ in reflect asm files.
Alternatively, runtime asm files still can be used to "implement"
functions in syscall package.

[brainman]

On Thursday, 21 August 2014 15:50:02 UTC+10, Dmitry Vyukov wrote:
> ... The better options is to keep that file in runtime, and use the asm

> thunks to call unexported runtime functions from syscall package.

That is what I am doing now. Thank you.

Alex

[minux]

On Aug 21, 2014 1:29 AM, "sanjay.m" <balas...@gmail.com> wrote:
> Thanks for the example; I am having the same problem with rdebug.goc (it declares symbols for runtime/debug, and uses runtime internal functions).
>
> To clarify, if the runtime-internal function is a Go function and the function take an argument, does the assembly stub still say "NOSPLIT,$0"? 

yes.

> I ask because the NOSPLIT docs seem to say that the NOSPLIT-ted function's frame needs space for all called functions, but I feel like that isn't true if a called function does the stack split check in its own preamble.

it's ok for nosplit function to split stack function.

[Rob Pike]

I think minux meant to write
It's OK for a NOSPLIT function to call a function that can trigger
a stack split.
The NOSPLIT marker refers only to the current frame.

By the way, a NOSPLIT function doesn't always have $0 as the last
element of its TEXT directive. It can't have locals (or the stack
would split) but it can have arguments, and the $X value should report
the size of the argument list.

-rob

[Dmitry Vyukov]

On Thu, Aug 21, 2014 at 6:24 PM, Rob Pike <r...@golang.org> wrote:
> I think minux meant to write
> It's OK for a NOSPLIT function to call a function that can trigger
> a stack split.
> The NOSPLIT marker refers only to the current frame.
>
> By the way, a NOSPLIT function doesn't always have $0 as the last
> element of its TEXT directive. It can't have locals (or the stack
> would split) but it can have arguments, and the $X value should report
> the size of the argument list.

A NOSPLIT function can have locals, just up to some limit. See e.g.
entersyscallblock in proc.c, it's NOSPLIT and has local vars.

[Dmitry Vyukov]

On Wed, Aug 20, 2014 at 9:31 PM, Keith Randall <k...@google.com> wrote:

>> Is there anything special about calling Go code on g0 stack long-term?
>
> I have a vision of a strict separation, Go code only on the G stack and C
> code only on the M stack (and a few assembly routines, like memmove, allowed
> on both). But that's harder and won't happen for 1.4. In particular the Go
> malloc code runs on the M stack right now.
>
> While there is Go code on the M stack, we do have to ensure that that Go
> code uses bounded stack.

Humm... I though that we want to get rid of C compiler altogether.

Then it does not need to be maintained, converted to Go and optimized.
I am pretty sure there is still a bunch codegen bugs. Instead of
giving me compilation errors, it silently crashes. Quality of code
that it generates is worse than Go already, and nobody is going to
improve it.

What are the reasons to have C code long-term?
If we want to do some optimizations based on g/g0 or otherwise
distinguish between them, we can have a //go:systemstack annotation
(that works only in runtime package).

Russ?

[Dmitry Vyukov]

Right now this decision affects me in the following way:

If I need to duplicate a C function in Go, is a temporal duplication or not?
Is there any value in converting to Go something that is not
performance critical and can perfectly live with onM switch?

[minux]

On Aug 21, 2014 10:44 AM, "'Dmitry Vyukov' via golang-dev" <golan...@googlegroups.com> wrote:
>
> On Wed, Aug 20, 2014 at 9:31 PM, Keith Randall <k...@google.com> wrote:
>
> >> Is there anything special about calling Go code on g0 stack long-term?
> >
> > I have a vision of a strict separation, Go code only on the G stack and C
> > code only on the M stack (and a few assembly routines, like memmove, allowed
> > on both).  But that's harder and won't happen for 1.4.  In particular the Go
> > malloc code runs on the M stack right now.
> >
> > While there is Go code on the M stack, we do have to ensure that that Go
> > code uses bounded stack.
>
> Humm... I though that we want to get rid of C compiler altogether.

i believe that's still the plan.

russ suggested that we might be able to use his c2go to convert the remaining c code, but we're not there yet.

[Keith Randall]

Yes, I believe the long term plan is to get rid of C altogether.  I wouldn't want to maintain duplicate code until that happens, though.

[Ian Lance Taylor]

On Thu, Aug 21, 2014 at 7:44 AM, 'Dmitry Vyukov' via golang-dev
<golan...@googlegroups.com> wrote:
>
> Humm... I though that we want to get rid of C compiler altogether.
>
> Then it does not need to be maintained, converted to Go and optimized.
> I am pretty sure there is still a bunch codegen bugs. Instead of
> giving me compilation errors, it silently crashes. Quality of code
> that it generates is worse than Go already, and nobody is going to
> improve it.
>
> What are the reasons to have C code long-term?
> If we want to do some optimizations based on g/g0 or otherwise
> distinguish between them, we can have a //go:systemstack annotation
> (that works only in runtime package).

When all C code lives on the M stack, we could reasonably consider
compiling it with GCC/Clang. I don't know if that is a good path to
follow, but the point is that the goal of removing the C compiler from
the sources is separable from the goal of converting all runtime code
to Go.

Ian

[Dmitry Vyukov]

---------- Forwarded message ----------
From: Russ Cox
Date: Wed, Feb 12, 2014 at 9:50 PM
Subject: Re: CL for growing stacks
To: Dmitry Vyukov

On Wed, Feb 12, 2014 at 12:49 PM, Dmitry Vyukov wrote:
>
> Compiling inner GC loop with gcc/clang would be awesome!

Except for having to build gcc/clang to build Go, and also having to
debug performance differences (or plain bugs) caused by different
people compiling the same Go version with different gcc/clang. Let's
not get distracted.

Russ

[zhuowe...@yahoo.com]

On Wednesday, August 20, 2014 12:01:24 AM UTC-7, Keith Randall wrote:

Removing a C function from the G stack can be done in a few ways:

I noticed that syscall_solaris.goc and syscall_windows.goc both call runtime.cgocall, which needs the currently executing g, so can't be executed on the m stack (using g0) - how should one convert them, then?

[Dmitry Vyukov]

You can get the currently running g on m stack from g->m->curg

[brainman]

I have some questions.

I tried to convert Syscall, but it calls runtime.cgocall, and runtime.cgocall is not converted to Go yet. Converting runtime.cgocall is, probably, too much for me - I am not familiar with that code at all. We would have to convert all cgo related code together.

Also converted Syscall becomes inefficient. It uses LibCall local variable, and I have to use new(libcall) for that, but that calls runtime.newobject. Also current Syscall uses parameters on stack as is by taking address of the first parameter. I could not do that in Go - it says "escapes to heap, not allowed in runtime." The only way to overcome that is to create another local array (another runtime.newobject call) and copy parameters there.

I can, probably, convert Syscall into asm, but it would still have to call runtime.cgocall, therefore it will not be NOSPLIT. Should I do that? I see no benefit.

Looking for suggestions. Thank you.

Alex

[Dmitry Vyukov]

On Wed, Aug 27, 2014 at 7:02 AM, brainman <alex.b...@gmail.com> wrote:
> I have some questions.
>
> I tried to convert Syscall, but it calls runtime.cgocall, and
> runtime.cgocall is not converted to Go yet. Converting runtime.cgocall is,
> probably, too much for me - I am not familiar with that code at all. We
> would have to convert all cgo related code together.

Start with just Syscall, we will convert cgocall later.

> Also converted Syscall becomes inefficient. It uses LibCall local variable,
> and I have to use new(libcall) for that, but that calls runtime.newobject.
> Also current Syscall uses parameters on stack as is by taking address of the
> first parameter. I could not do that in Go - it says "escapes to heap, not
> allowed in runtime." The only way to overcome that is to create another
> local array (another runtime.newobject call) and copy parameters there.
>
> I can, probably, convert Syscall into asm, but it would still have to call
> runtime.cgocall, therefore it will not be NOSPLIT. Should I do that? I see
> no benefit.
>
> Looking for suggestions. Thank you.

Declare the function that accepts the pointer to LibCall as //go:noescape.
See example below, I allocate keventstruct on stack and pass its
address to syscall:
https://codereview.appspot.com/132910043/diff/140001/src/pkg/runtime/netpoll_kqueue.go

[Aram Hăvărneanu]

I assigned myself to syscall_solaris.goc.

Why are some solaris functions from other files missing, say functions
from os_solaris.c?

--
Aram Hăvărneanu

[Dmitry Vyukov]

On Wed, Aug 27, 2014 at 4:33 PM, Aram Hăvărneanu <ara...@mgk.ro> wrote:
> I assigned myself to syscall_solaris.goc.
>
> Why are some solaris functions from other files missing, say functions
> from os_solaris.c?

I suspect Keith looked at just .goc files. The ultimate criteria -- is
it running on g stack? If yes, then it must be converted. If it runs
only on system g0 stack, then it does not need to be converted right
now.

[Aram Hăvărneanu]

Is it possible to call a variadic C function from Go?

--
Aram Hăvărneanu

[Ian Lance Taylor]

On Thu, Aug 28, 2014 at 8:27 AM, Aram Hăvărneanu <ara...@mgk.ro> wrote:
>
> Is it possible to call a variadic C function from Go?

It should work fine in the runtime package, except that the Go code
will not be able to see any return value.

Ian

[Dmitry Vyukov]

If you want to convert solaris syscalls, then you can check out what I
am doing for windows' stdcall:
https://codereview.appspot.com/135090043
It's not yet done completely, and it was not reviewed, but it should work.