SoFunction
Updated on 2025-03-02

Detailed explanation of collaboration and preemption in Go scheduler learning

0. Introduction

In the previous blog—"Golang Scheduler (4)—goroutine Scheduler"There has been an unanswered question left in this: If a G executes too long, how can other Gs be scheduled normally, which leads to the next topic:Collaboration and preemption。

In Gov1.2The version implements preemptive calls based on collaboration. The basic principle of this call is:

  • whensysmonIf the monitoring thread finds that the execution time of a coroutine is too long, it will friendlyly set preemption marks for the coroutine;
  • When this coroutineCallWhen a function is used, it will check whether the stack is expanded, and here it will check for preemption marks. If they are marked, the CPU will be given up to achieve scheduling.

However, this scheduling method is proactive and based on collaboration, but it cannot face some scenarios. For example, if no call occurs in the dead loop, then the coroutine will be executed forever and scheduling will never occur, which is obviously unacceptable.

So, inv1.14In version, Go finally introduced signal-based preemption scheduling. Below, we will introduce these two preemption scheduling.

1. The user takes the initiative to give up the CPU: function

Before introducing the two preemption scheduling, let's first introducefunction:

// Gosched yields the processor, allowing other goroutines to run. It does not
// suspend the current goroutine, so execution resumes automatically.
func Gosched() {
   checkTimeouts()
   mcall(gosched_m)
}

According to the instructions,The function will actively abandon the current processor and allow other coroutines to execute, but will not pause itself, but will only transfer the scheduling rights, and then rely on the scheduler to obtain rescheduling.

After that, it will passmcallSwitch function tog0To execute the stackgosched_mfunction:

// Gosched continuation on g0.
func gosched_m(gp *g) {
   if  {
      traceGoSched()
   }
   goschedImpl(gp)
}

gosched_mCallgoschedImplfunction, which will be coroutinegpTransfer this M andgpPut it in the global queue and wait for scheduling.

func goschedImpl(gp *g) {
   status := readgstatus(gp)
   if status&^_Gscan != _Grunning {
      dumpgstatus(gp)
      throw("bad g status")
   }
   casgstatus(gp, _Grunning, _Grunnable)
   dropg()            // Make the current m give up gp, which is its parameter curg   lock(&)
   globrunqput(gp)    // And put gp into the global queue and wait for scheduling   unlock(&)

   schedule()
}

AlthoughIt has the ability to actively give up the CPU, but it has high requirements for users and is not user-friendly.

2. Preemptive scheduling based on collaboration

2.1 Scenario

package main

import (
   "fmt"
   "runtime"
   "sync"
   "time"
)

var once = {}

func f() {
   (func() {
      ("I am go routine 1!")
   })
}

func main() {
   defer ((1))

   go func() {
      for {
         f()
      }
   }()

   (10 * )
   ("I am main goroutine!")
}

We consider the above code. First, we set the number of P to 1, and then start a coroutine to enter a dead loop, and call a function in a loop. If the dispatch is not preempted, the coroutine will always occupy P, that is, it will always occupy the CPU, and the code will never be executed("I am main goroutine!")This is the job. Let’s take a look at how collaborative preemption avoids the above problems.

2.2 Stack expansion and preemption mark

$ go tool compile -N -l
$ go tool objdump >>

Through the above instructions, we obtain the assembly code of the code in 2.1 and intercept thefThe assembly code of the function is as follows:

TEXT "".f(SB) gofile../home/chenyiguo/smb_share/go_routine_test/
  :12      0x151a       493b6610      CMPQ 0x10(R14), SP 
  :12      0x151e       762b         JBE 0x154b    
  :12      0x1520       4883ec18      SUBQ $0x18, SP    
  :12      0x1524       48896c2410    MOVQ BP, 0x10(SP)  
  :12      0x1529       488d6c2410    LEAQ 0x10(SP), BP  
  :13      0x152e       488d0500000000    LEAQ 0(IP), AX    [3:7]R_PCREL:"".once      
  :13      0x1535       488d1d00000000    LEAQ 0(IP), BX    [3:7]R_PCREL:"".f.func1·f  
  :13      0x153c       e800000000    CALL 0x1541       [1:5]R_CALL:sync.(*Once).Do    
  :16      0x1541       488b6c2410    MOVQ 0x10(SP), BP  
  :16      0x1546       4883c418      ADDQ $0x18, SP    
  :16      0x154a       c3       RET          
  :12      0x154b       e800000000    CALL 0x1550       [1:5]R_CALL:runtime.morestack_noctxt   
  :12      0x1550       ebc8         JMP "".f(SB)  

The first line,CMPQ 0x10(R14), SPJust comparisonSPand0x10(R14)(Actuallystackguard0) size (noteAT&TIn formatCMPThe order of series instructions) whenSPLess than or equal to0x10(R14)When it is transferred to0x154bAddress callruntime.morestack_noctxt, trigger the stack expansion operation. In fact, if you observe carefully, you will find that all the functions' prologues (the front of the function call) are inserted into the detection instructions unless marked on the function.//go:nosplit。

Next, we will focus on two points to open up the entire link, namely:

  • How to reschedule stack expansion and give up the CPU's execution rights?
  • When will the stack expansion mark be set?

2.3 How to trigger rescheduling when stack expansion

// morestack but not preserving ctxt.
TEXT runtime·morestack_noctxt(SB),NOSPLIT,$0
   MOVL   $0, DX
   JMP    runtime·morestack(SB)

TEXT runtime·morestack(SB),NOSPLIT,$0-0
   ...

   // Set g->sched to context in f.
   MOVQ   0(SP), AX // f's PC
   MOVQ   AX, (g_sched+gobuf_pc)(SI)
   LEAQ   8(SP), AX // f's SP
   MOVQ   AX, (g_sched+gobuf_sp)(SI)
   MOVQ   BP, (g_sched+gobuf_bp)(SI)
   MOVQ   DX, (g_sched+gobuf_ctxt)(SI)

   ...
   CALL   runtime·newstack(SB)
   CALL   runtime·abort(SB)  // crash if newstack returns
   RET

In the above code,runtime·morestack_noctxtCallruntime·morestack,existruntime·morestackIn the coroutine, the PC and SP of the coroutine will be recorded first, and then the：

func newstack() {
   ...

   gp := 
   
   ...
   stackguard0 := (&gp.stackguard0)

   ...
   preempt := stackguard0 == stackPreempt
   ...

   if preempt {
      if gp == .g0 {
         throw("runtime: preempt g0")
      }
      if  == 0 &&  == 0 {
         throw("runtime: g is running but p is not")
      }

      if  {
         // We're at a synchronous safe point now, so
         // do the pending stack shrink.
          = false
         shrinkstack(gp)
      }

      if  {
         preemptPark(gp) // never returns
      }

      // Act like goroutine called .
      gopreempt_m(gp) // never return
   }

   ...
}

We simplifyFunctions, summarizing it, are through existing work coroutinesstackguard0Fields to determine whether preemption should occur. If necessary, callgopreempt_m(gp)function:

func gopreempt_m(gp *g) {
   if  {
      traceGoPreempt()
   }
   goschedImpl(gp)
}

You can see,gopreempt_mFunctions and mentioned earlierGoschedWhat is mentioned in the functiongosched_mLike functions, they will be calledgoschedImplFunction, for coroutinesgpTransfer this M andgpPut it in the global queue and wait for scheduling.

Here we understand that once stack expansion occurs, there is a possibility of transferring the execution rights and rescheduling. So when will stack expansion occur?

2.4 When to set the stack expansion mark

In the code,stackguard0Set field tostackPreemptThere are many places, but the one that matches our above scenario is still the background monitoring threadsysmonLoop, for those stuck in system calls and long-runninggoroutineThe right to run is seizedretakefunction:

func sysmon() {
   ...

   for {
      ...
      // retake P's blocked in syscalls
      // and preempt long running G's
      if retake(now) != 0 {
         idle = 0
      } else {
         idle++
      }
      ...
   }
}
func retake(now int64) uint32 {
   ...
   for i := 0; i < len(allp); i++ {
      ...
      s := _p_.status
      sysretake := false
      if s == _Prunning || s == _Psyscall {
         // Preempt G if it's running for too long.
         t := int64(_p_.schedtick)
         if int64() != t {
             = uint32(t)
             = now
         } else if +forcePreemptNS <= now { // forcePreemptNS=10ms
            preemptone(_p_) // Set the stack expansion mark here            // In case of syscall, preemptone() doesn't
            // work, because there is no M wired to P.
            sysretake = true
         }
      }
      ...
   }
   unlock(&allpLock)
   return uint32(n)
}

Among them, inpreemptoneSettings for stack expansion marks in the function:

func preemptone(_p_ *p) bool {
   mp := _p_.()
   if mp == nil || mp == getg().m {
      return false
   }
   gp := 
   if gp == nil || gp == mp.g0 {
      return false
   }

    = true

   // Every call in a goroutine checks for stack overflow by
   // comparing the current stack pointer to gp->stackguard0.
   // Setting gp->stackguard0 to StackPreempt folds
   // preemption into the normal stack overflow check.
   gp.stackguard0 = stackPreempt // Set stack expansion mark
   // Request an async preemption of this P.
   if preemptMSupported &&  == 0 {
      _p_.preempt = true
      preemptM(mp)
   }

   return true
}

Through the above, we colludedgoroutineThe logic of collaborative preemption:

  • First of all, the background monitoring thread will run too long (≥10ms) coroutine sets stack expansion marks;
  • When a coroutine runs to the prologue of any function, it will first check the stack expansion mark;
  • If stack expansion is required, the operation rights of this coroutine will be seized when stack expansion is performed, thereby realizing preemptive scheduling.

3. Signal-based preemptive scheduling

Analyzing the above conclusions, we can know that the above preemption trigger logic has a fatal disadvantage, that is, it must be run to the preamble part of the function stack, and this cannot read the running rights of the following coroutines. Before Go version 1.14, the code will not print the last sentence."I am main goroutine!"：

package main

import (
   "fmt"
   "runtime"
   "sync"
   "time"
)

var once = {}

func main() {
   defer ((1))

   go func() {
      for {
         (func() {
            ("I am go routine 1!")
         })
      }
   }()

   (10 * )
   ("I am main goroutine!")
}

Because in the above coroutineforA loop is a dead loop and does not contain stack expansion logic, so its own execution rights will not be transferred.

3.1 Send preemption signal

to this end,Go SDKSignal-based preemptive scheduling is introduced. We pay attention to the analysis of the previous sectionpreemptoneThe function code has the following parts:

if preemptMSupported &&  == 0 {
   _p_.preempt = true
   preemptM(mp)
}

inpreemptMThe function will be sent_SIGURGSignals to threads that need to be preempted:

const sigPreempt = _SIGURG


func preemptM(mp *m) {
   // On Darwin, don't try to preempt threads during exec.
   // Issue #41702.
   if GOOS == "darwin" || GOOS == "ios" {
      ()
   }

   if (&, 0, 1) {
      if GOOS == "darwin" || GOOS == "ios" {
         (&pendingPreemptSignals, 1)
      }

      // If multiple threads are preempting the same M, it may send many
      // signals to the same M such that it hardly make progress, causing
      // live-lock problem. Apparently this could happen on darwin. See
      // issue #37741.
      // Only send a signal if there isn't already one pending.
      signalM(mp, sigPreempt)
   }

   if GOOS == "darwin" || GOOS == "ios" {
      ()
   }
}

3.2 Preemption call injection

Speaking of this, we need to go back to the beginning, in the first coroutinem0OpenmstartOn the calling link, it will be calledmstartm0Functions will be called hereinitsig：

func initsig(preinit bool) {
  ...

   for i := uint32(0); i < _NSIG; i++ {
      ...

      handlingSig[i] = 1
      setsig(i, (sighandler))
   }
}

In the above, registeredsighandlerfunction:

func sighandler(sig uint32, info *siginfo, ctxt , gp *g) {
   ...

   if sig == sigPreempt &&  == 0 {
      // Might be a preemption signal.
      doSigPreempt(gp, c)
      // Even if this was definitely a preemption signal, it
      // may have been coalesced with another signal, so we
      // still let it through to the application.
   }

   ...
}

Then receivesigPreemptWhen the signal is passeddoSigPreemptFunction processing is as follows:

func doSigPreempt(gp *g, ctxt *sigctxt) {
   // Check if this G wants to be preempted and is safe to
   // preempt.
   if wantAsyncPreempt(gp) {
      if ok, newpc := isAsyncSafePoint(gp, (), (), ()); ok {
         // Adjust the PC and inject a call to asyncPreempt.
         (abi.FuncPCABI0(asyncPreempt), newpc) // Insert preemption call      }
   }

   // Acknowledge the preemption.
   (&amp;, 1)
   (&amp;, 0)

   if GOOS == "darwin" || GOOS == "ios" {
      (&amp;pendingPreemptSignals, -1)
   }
}

final,doSigPreempt—>asyncPreempt->asyncPreempt2：

func asyncPreempt2() {
   gp := getg()
    = true
   if  {
      mcall(preemptPark)
   } else {
      mcall(gopreempt_m)
   }
    = false
}

Then, we're back to what we're familiar withgopreempt_mFunctions, I won't go into details here.

Therefore, for signal-based preemption scheduling, the summary is as follows:

  • M1 sends a signal_SIGURG；
  • M2 receives the signal and processes it through the signal processing function;
  • M2 modifies the execution context and restores to the modified location;
  • Re-enter the scheduling loop and then schedule othergoroutine。

4. Summary

Overall,GoThe development of scheduling strategies is also gradually developing with the enrichment of demand. Collaborative scheduling can ensure that users with function calls can stop normally; preemptive scheduling can avoid garbage collection delays at any time caused by the dead loop.

This is the end of this article about the detailed explanation of collaboration and preemption of Go scheduler learning. For more related Go scheduler content, please search for my previous articles or continue browsing the related articles below. I hope everyone will support me in the future!