Why runtime.sigtrampgo calls runtime.morestack?

[changkun]

As we all know that a program is interrupted by a signal and entering kernel space then switch to a userspace signal handler. After the signal handler is accomplished, it will be reentering the kernel space then switch back to where was interrupted.

I am recently reading the newly implemented async preemption in go 1.14, which uses the OS signal to interrupt a "non-preemptive" user goroutine. I am debugging very simple program:

    package main


    import (
        "runtime"
        "time"
    )


    func tightloop() {
        for {
        }
    }


    func main() {
        runtime.GOMAXPROCS(1)
        go tightloop()


        time.Sleep(time.Millisecond)
        println("OK")
        runtime.Gosched()
    }

In Go 1.14, when a preempt signal arrives, the `tightloop` will be interrupted by the OS and entering the pre-configured signal handler `runtime·sigtramp`:

    TEXT runtime·sigtramp(SB),NOSPLIT,$72
        MOVQ DX, ctx-56(SP)
        MOVQ SI, info-64(SP)
        MOVQ DI, signum-72(SP)
        MOVQ $runtime·sigtrampgo(SB), AX
        CALL AX
        RET

which `sigtrampgo` eventually calls the `sighandler`.

    //go:nosplit
    //go:nowritebarrierrec
    func sigtrampgo(sig uint32, info *siginfo, ctx unsafe.Pointer) {
        (...)
        setg(g.m.gsignal)
        (...)
        sighandler(sig, info, ctx, g)
        setg(g)
        (...)
    }

As far as I read the `sighandler` function, it calls `doSigPreempt` and modifies the `ctx` that passed from system kernel, and sets the `rip` to the prologue of `runtime.asyncPreempt`.

    //go:nowritebarrierrec
    func sighandler(sig uint32, info *siginfo, ctxt unsafe.Pointer, gp *g) {
        _g_ := getg()
        c := &sigctxt{info, ctxt}


        (...)
        if sig == sigPreempt {
            doSigPreempt(gp, c)
        }
    }
    func doSigPreempt(gp *g, ctxt *sigctxt) {
        if canPreempt {
            // here modifies the rip and rsp
            ctxt.pushCall(funcPC(asyncPreempt))
        }


        (...)
    }

However, I noticed that the asyncPreempt is not immediately executed when

the signal handler is complete, instead:

1. `morestack` or `morestack_noctxt` is called after `sighandler` is **returned** (not entering either the epilogue or prologue), which calls `newstack` and check checks the preempt flag and entering schedule loop and therefore schedules the main goroutine to finish the async preemption.

2. the `OK` outputs before executing `asyncPreempt`

Here are my inserted print logs in runtime:

    mstart1 call schedule()
    enter schedule()
    park_m call schedule()
    enter schedule()
    mstart1 call schedule()
    enter schedule()
    mstart1 call schedule()
    enter schedule()
    park_m call schedule()
    enter schedule()
    park_m call schedule()
    enter schedule()
    park_m call schedule()
    enter schedule()
    mstart1 call schedule()
    enter schedule()
    park_m call schedule()
    enter schedule()
    rip: 17149264 eip: 824634034136
    before pushCall asyncPreempt
    after pushCall asyncPreempt
    rip: 17124704 eip: 824634034128      // rip points to asyncPreempt
    calling newstack: m0, g0             // how could newstack is called?
    newstack call gopreempt_m
    gopreempt_m call goschedImpl
    goschedImpl call schedule()
    enter schedule()
    OK
    gosched_m call goschedImpl
    goschedImpl call schedule()
    enter schedule()
    asyncPreempt2
    asyncPreempt2
    asyncPreempt2
    asyncPreempt2
    preemptPark
    gopreempt_m call goschedImpl
    goschedImpl call schedule()
    enter schedule()

while I checked the dumped assembly code, there is no stack split check

in neither `asyncPreempt` or `sigtramp`.

Sorry for the long story, my questions are:

- When, who, and how runtime calls the `morestack` after `sighandler`? What did I miss?

- Does modifying `ctx` changes program jumps to the modified `rip` instruction after finishing the signal handler?

Thank you very much for reading the question and thanks to the go team building such a brilliant feature.

[Ian Lance Taylor]

It sounds like you are doing a good job of investigating. I don't
know the answer to your question, but it sounds worth figuring out. I
would encourage you to keep looking into this. Be sure to track which
goroutine is seeing the preemption call; perhaps there is some
confusion between different goroutines. Or, in newstack you could
perhaps call getcallerpc to find out exactly what is calling the
function.

Ian

[changkun]

Dear Ian,

Thank you so much for your hint, I think I've figured it out.

The root cause seems similar to the "uncertainty principle".

As an observer, by adding a `println` call in `asyncPreempt` 

as well as `asyncPreempt2` influences the actual behavior 

after signal handling. The `println` involves stack split check, 

which calls the `morestack`.

It took me a while to realize that `morestack` stores its caller 

pc in `g.m.morebuf.pc` since `getcallerpc` in `newstack` 

always returns the pc from `morestack`, which doesn't tell 

too much information.

//go:nosplit
func asyncPreempt2() {
 // println("asyncPreempt2 is called") // comment here omits calling morestack.
 gp := getg()
 gp.asyncSafePoint = true
 if gp.preemptStop {
 mcall(preemptPark)
 } else {
 mcall(gopreempt_m)
 }
 println("asyncPreempt2 finished")
 gp.asyncSafePoint = false
}

[Ian Lance Taylor]

On Sun, Dec 8, 2019 at 1:10 PM changkun <euryu...@gmail.com> wrote:
>
> Thank you so much for your hint, I think I've figured it out.
> The root cause seems similar to the "uncertainty principle".
>
> As an observer, by adding a `println` call in `asyncPreempt`
> as well as `asyncPreempt2` influences the actual behavior
> after signal handling. The `println` involves stack split check,
> which calls the `morestack`.

Ah, of course. One workaround is to use write, as in

// Package-scope variable
var myMsg = []byte("my message")

// In function.
write(2, unsafe.Pointer(&myMsg[0]), int32(len(myMsg)))

This works because the various implementations of write are marked nosplit.

Ian

[changkun]

That's a really useful workaround! It makes much more sense now.

Conclusion: when `sigtramp` is returned, nothing is called before entering `ayncPreempt`.

Thank you so much for releasing me from confusing :)

sigtramp is returned
asyncPreempt is called
OK

 CALL runtime·sigtrampgo(SB)
 CALL runtime·printmsg(SB)


TEXT ·asyncPreempt(SB),NOSPLIT|NOFRAME,$0-0
 CALL runtime·printmsg2(SB)


var msg1 = []byte("sigtramp is returned\n")
var msg2 = []byte("asyncPreempt is called\n")


//go:nosplit
func printmsg() {
 write(2, unsafe.Pointer(&msg1[0]), int32(len(msg1)))
}


//go:nosplit
func printmsg2() {
 write(2, unsafe.Pointer(&msg2[0]), int32(len(msg2)))
}