[go] runtime: enable race detector for atomic operations on linux/ppc64le
7 views
Skip to first unread message
Jayanth Krishnamurthy (Gerrit)
Nov 6, 2025, 3:49:37 PM (6 days ago) Nov 6
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Jayanth Krishnamurthy has uploaded the change for review![]( "Open in Gerrit")
Commit message
runtime: enable race detector for atomic operations on linux/ppc64le
This CL completes the race detector implementation for linux/ppc64le by
adding proper TSAN instrumentation and real atomic LL/SC sequences for
sync/atomic operations.
Previously, the ppc64le race wrappers invoked the __tsan_go_atomic*
hooks but never executed the actual LL/SC (load-reserve/store-conditional)
instructions after instrumentation. As a result, atomic operations such as
CompareAndSwap* and And*/Or* either hung or returned incorrect results when
built with -race.
This fixes:
1. And/Or wrappers: Set up the correct argument pointer (R6) so that
racecallatomic can locate the target address. Without this, And/Or
operations could spin indefinitely on failed CAS loops.
2. CompareAndSwap wrappers: Replace BR (tail call) with BL (branch-link)
so execution returns after TSAN instrumentation and proceeds to the
actual atomic LL/SC sequence. Previously, BR prevented CAS execution.
3. racecallatomic helper: Add POWER-specific validation guards
- Nil address check
- 4-byte alignment check (required by LL/SC)
- Heap arena range validation [racearenastart, racearenaend)
- Racedata range validation [racedatastart, racedataend)
- Runtime state checks (g, racectx) to avoid early initialization crashes
4. CAS loops: Full 32-bit and 64-bit implementations using
- LWAR/STWCCC and LDAR/STDCCC for LL/SC semantics
- SYNC (before CAS) and ISYNC (after CAS) for sequential consistency
The alignment check is specific to the POWER architecture. The racectx validation prevents early runtime
faults when the TSAN context is not yet initialized.
Testing
-------
Validated on POWER9 and POWER10 (ppc64le):
Go core
- sync/atomic: 92/93 tests complete (99%)
- TestValueConcurrent flags a true race in math/rand test code (expected)
- runtime/race: PASS
- ./all.bash: passes except for known intentional race tests like
- net.TestFileCloseRace
- os.TestClosedPipeRaceRead / Write
- os.TestFdReadRace
- os/exec.TestEchoFileRace
Kubernetes tests (they failed before with -race)
- k8s.io/kubernetes/pkg/controller/devicetainteviction:
- Normal: 37.63 s
- With -race: 163.84 s (4.4x)
Race-enabled builds are showing a 4-5x slowdown on real workloads.
Fixes #76051
Change-Id: Iaf578449a6171a0c6f7c33ec6f64c1251297ae6d
Cq-Include-Trybots: luci.golang.try:gotip-linux-ppc64_power10,gotip-linux-ppc64_power8,gotip-linux-ppc64le_power8,gotip-linux-ppc64le_power9,gotip-linux-ppc64le_power10
Change diff
diff --git a/src/runtime/race_ppc64le.s b/src/runtime/race_ppc64le.s
index b327e49..ebde463 100644
--- a/src/runtime/race_ppc64le.s
+++ b/src/runtime/race_ppc64le.s
@@ -329,11 +329,13 @@
TEXT sync∕atomic·AndInt32(SB), NOSPLIT, $0-20
GO_ARGS
MOVD $__tsan_go_atomic32_fetch_and(SB), R8
+ ADD $32, R1, R6
BR racecallatomic<>(SB)
TEXT sync∕atomic·AndInt64(SB), NOSPLIT, $0-24
GO_ARGS
MOVD $__tsan_go_atomic64_fetch_and(SB), R8
+ ADD $32, R1, R6
BR racecallatomic<>(SB)
TEXT sync∕atomic·AndUint32(SB), NOSPLIT, $0-20
@@ -352,11 +354,13 @@
TEXT sync∕atomic·OrInt32(SB), NOSPLIT, $0-20
GO_ARGS
MOVD $__tsan_go_atomic32_fetch_or(SB), R8
+ ADD $32, R1, R6
BR racecallatomic<>(SB)
TEXT sync∕atomic·OrInt64(SB), NOSPLIT, $0-24
GO_ARGS
MOVD $__tsan_go_atomic64_fetch_or(SB), R8
+ ADD $32, R1, R6
BR racecallatomic<>(SB)
TEXT sync∕atomic·OrUint32(SB), NOSPLIT, $0-20
@@ -378,15 +382,59 @@
// ThreadState *thr, uptr cpc, uptr pc, u8 *a)
MOVD $__tsan_go_atomic32_compare_exchange(SB), R8
ADD $32, R1, R6 // addr of caller's 1st arg
- BR racecallatomic<>(SB)
+ BL racecallatomic<>(SB)
+ //CAS
+ MOVD addr+0(FP), R3
+ MOVW old+8(FP), R4
+ MOVW new+12(FP), R5
+
+ SYNC
+cas32_loop:
+ LWAR (R3), R7
+ CMPW R7, R4
+ BNE cas32_fail
+ STWCCC R5, (R3)
+ BNE cas32_loop
+ ISYNC
+ MOVD $1, R6
+ MOVB R6, ret+16(FP)
+ RET
+cas32_fail:
+ ISYNC
+ MOVD $0, R6
+ MOVB R6, ret+16(FP)
+ RET
+
TEXT sync∕atomic·CompareAndSwapInt64(SB), NOSPLIT, $0-25
GO_ARGS
// void __tsan_go_atomic32_compare_exchange(
// ThreadState *thr, uptr cpc, uptr pc, u8 *a)
MOVD $__tsan_go_atomic64_compare_exchange(SB), R8
ADD $32, R1, R6 // addr of caller's 1st arg
- BR racecallatomic<>(SB)
+ BL racecallatomic<>(SB)
+
+ // CAS
+ MOVD addr+0(FP), R3
+ MOVD old+8(FP), R4
+ MOVD new+16(FP), R5
+
+ SYNC
+cas64_loop:
+ LDAR (R3), R7
+ CMP R7, R4
+ BNE cas64_fail
+ STDCCC R5, (R3)
+ BNE cas64_loop
+ ISYNC
+ MOVD $1, R6
+ MOVB R6, ret+24(FP)
+ RET
+cas64_fail:
+ ISYNC
+ MOVD $0, R6
+ MOVB R6, ret+24(FP)
+ RET
TEXT sync∕atomic·CompareAndSwapUint32(SB), NOSPLIT, $0-17
GO_ARGS
@@ -407,60 +455,46 @@
// R6 = addr of incoming arg list
// R8 contains addr of target function.
TEXT racecallatomic<>(SB), NOSPLIT, $0-0
- // Trigger SIGSEGV early if address passed to atomic function is bad.
- MOVD (R6), R7 // 1st arg is addr
- MOVB (R7), R9 // segv here if addr is bad
- // Check that addr is within [arenastart, arenaend) or within [racedatastart, racedataend).
- MOVD runtime·racearenastart(SB), R9
- CMP R7, R9
- BLT racecallatomic_data
- MOVD runtime·racearenaend(SB), R9
- CMP R7, R9
- BLT racecallatomic_ok
-racecallatomic_data:
+ // 1. target
+ MOVD 0(R6), R7
+ CMP R7, $0
+ BEQ race_ret
+
+ // 2. alignment
+ ANDCC $3, R7, R11
+ BNE race_ret
+
+ // 3. heap arena?
+ MOVD runtime·racearenastart(SB), R9
+ MOVD runtime·racearenaend(SB), R10
+ CMPU R7, R9
+ BLT check_racedata
+ CMPU R7, R10
+ BGE check_racedata
+ BR do_racecall
+
+check_racedata:
+ // 4. racedata range
MOVD runtime·racedatastart(SB), R9
- CMP R7, R9
- BLT racecallatomic_ignore
- MOVD runtime·racedataend(SB), R9
- CMP R7, R9
- BGE racecallatomic_ignore
-racecallatomic_ok:
- // Addr is within the good range, call the atomic function.
- MOVD runtime·tls_g(SB), R10
- MOVD 0(R10), g
- MOVD g_racectx(g), R3 // goroutine racectx aka *ThreadState
- MOVD R8, R5 // pc is the function called
- MOVD (R1), R4 // caller pc from stack
- BL racecall<>(SB) // BL needed to maintain stack consistency
- RET //
-racecallatomic_ignore:
- // Addr is outside the good range.
- // Call __tsan_go_ignore_sync_begin to ignore synchronization during the atomic op.
- // An attempt to synchronize on the address would cause crash.
- MOVD R8, R15 // save the original function
- MOVD R6, R17 // save the original arg list addr
- MOVD $__tsan_go_ignore_sync_begin(SB), R8 // func addr to call
- MOVD runtime·tls_g(SB), R10
- MOVD 0(R10), g
- MOVD g_racectx(g), R3 // goroutine context
- BL racecall<>(SB)
- MOVD R15, R8 // restore the original function
- MOVD R17, R6 // restore arg list addr
- // Call the atomic function.
- // racecall will call LLVM race code which might clobber r30 (g)
- MOVD runtime·tls_g(SB), R10
- MOVD 0(R10), g
+ MOVD runtime·racedataend(SB), R10
+ CMPU R7, R9
+ BLT race_ret
+ CMPU R7, R10
+ BGE race_ret
- MOVD g_racectx(g), R3
- MOVD R8, R4 // pc being called same TODO as above
- MOVD (R1), R5 // caller pc from latest LR
- BL racecall<>(SB)
- // Call __tsan_go_ignore_sync_end.
- MOVD $__tsan_go_ignore_sync_end(SB), R8
- MOVD g_racectx(g), R3 // goroutine context g should still be good?
- BL racecall<>(SB)
- RET
+do_racecall:
+ // Set up racectx (R3), caller PC (R4), and target hook (R5)
+ MOVD runtime·tls_g(SB), R10
+ MOVD 0(R10), g
+ MOVD g_racectx(g), R3
+ CMP R3, $0
+ BEQ race_ret // if no racectx yet, skip instrumentation
+ MOVD 0(R1), R4 // caller PC
+ OR R8, R8, R5 // R5 = TSAN hook (from R8)
+ BL racecall<>(SB) // call TSAN wrapper (preserves stack linkage)
+race_ret:
+ RET
// void runtime·racecall(void(*f)(...), ...)
// Calls C function f from race runtime and passes up to 4 arguments to it.
// The arguments are never heap-object-preserving pointers, so we pretend there are no arguments.
Change information
Files:
- M src/runtime/race_ppc64le.s
Change size: M
Delta: 1 file changed, 87 insertions(+), 53 deletions(-)
Related details
Attention set is empty
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Jayanth Krishnamurthy (Gerrit)
Nov 6, 2025, 3:50:42 PM (6 days ago) Nov 6
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Jayanth Krishnamurthy uploaded new patchset![]( "Open in Gerrit")
Jayanth Krishnamurthy uploaded patch set #2 to this change.
Related details
Attention set is empty
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Jayanth Krishnamurthy (Gerrit)
Nov 6, 2025, 3:51:12 PM (6 days ago) Nov 6
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Jayanth Krishnamurthy uploaded new patchset![]( "Open in Gerrit")
Jayanth Krishnamurthy uploaded patch set #3 to this change.
Related details
Attention set is empty
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Jayanth Krishnamurthy (Gerrit)
Nov 6, 2025, 3:59:30 PM (6 days ago) Nov 6
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Jayanth Krishnamurthy uploaded new patchset![]( "Open in Gerrit")
Jayanth Krishnamurthy uploaded patch set #4 to this change.
Related details
Attention set is empty
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Jayanth Krishnamurthy (Gerrit)
Nov 6, 2025, 4:01:48 PM (6 days ago) Nov 6
to goph...@pubsubhelper.golang.org, Paul Murphy, Austin Clements, Keith Randall, Martin Möhrmann, Gopher Robot, golang-co...@googlegroups.com
Attention needed from Austin Clements, Keith Randall, Martin Möhrmann and Paul Murphy
Jayanth Krishnamurthy voted Commit-Queue+1![]( "Open in Gerrit")
| Commit-Queue | +1 |
Related details
Attention is currently required from:
- Austin Clements
- Keith Randall
- Martin Möhrmann
- Paul Murphy
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Jayanth Krishnamurthy (Gerrit)
Nov 7, 2025, 3:23:40 AM (6 days ago) Nov 7
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention needed from Austin Clements, Jayanth Krishnamurthy, Keith Randall, Martin Möhrmann and Paul Murphy
Jayanth Krishnamurthy uploaded new patchset![]( "Open in Gerrit")
Attention is currently required from:
- Austin Clements
- Jayanth Krishnamurthy
- Keith Randall
- Martin Möhrmann
- Paul Murphy
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Jayanth Krishnamurthy (Gerrit)
Nov 7, 2025, 3:45:42 AM (6 days ago) Nov 7
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention needed from Austin Clements, Jayanth Krishnamurthy, Keith Randall, Martin Möhrmann and Paul Murphy
Jayanth Krishnamurthy uploaded new patchset![]( "Open in Gerrit")
Jayanth Krishnamurthy uploaded patch set #6 to this change.
Related details
Attention is currently required from:
- Austin Clements
- Jayanth Krishnamurthy
- Keith Randall
- Martin Möhrmann
- Paul Murphy
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Jayanth Krishnamurthy (Gerrit)
Nov 7, 2025, 3:47:12 AM (6 days ago) Nov 7
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention needed from Austin Clements, Jayanth Krishnamurthy, Keith Randall, Martin Möhrmann and Paul Murphy
Jayanth Krishnamurthy uploaded new patchset![]( "Open in Gerrit")
Jayanth Krishnamurthy uploaded patch set #7 to this change.
Related details
Attention is currently required from:
- Austin Clements
- Jayanth Krishnamurthy
- Keith Randall
- Martin Möhrmann
- Paul Murphy
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Jayanth Krishnamurthy (Gerrit)
Nov 7, 2025, 3:47:49 AM (6 days ago) Nov 7
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention needed from Austin Clements, Jayanth Krishnamurthy, Keith Randall, Martin Möhrmann and Paul Murphy
Jayanth Krishnamurthy uploaded new patchset![]( "Open in Gerrit")
Jayanth Krishnamurthy uploaded patch set #8 to this change.
Related details
Attention is currently required from:
- Austin Clements
- Jayanth Krishnamurthy
- Keith Randall
- Martin Möhrmann
- Paul Murphy
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Paul Murphy (Gerrit)
Nov 11, 2025, 4:34:55 PM (2 days ago) Nov 11
to Jayanth Krishnamurthy, goph...@pubsubhelper.golang.org, Go LUCI, Austin Clements, Keith Randall, Martin Möhrmann, Gopher Robot, golang-co...@googlegroups.com
Attention needed from Austin Clements, Jayanth Krishnamurthy, Keith Randall and Martin Möhrmann
Paul Murphy added 4 comments![]( "Open in Gerrit")
File src/runtime/race_ppc64le.s
Line 388, Patchset 8 (Latest):
MOVD addr+0(FP), R3Paul Murphy . unresolved
I'm not sure these are correct? No other implementation does this, and the tsan performs this op.
Line 455, Patchset 8 (Latest):
// R6 = addr of incoming arg listPaul Murphy . unresolved
Should R7 be mentioned here?
Line 464, Patchset 8 (Latest):
ANDCC $3, R7, R11Paul Murphy . unresolved
This doesn't seem right. If this is the argument pointer, isn't it aligned to the type which isn't necessarily 64b?
Line 496, Patchset 8 (Latest):
race_ret:Paul Murphy . unresolved
Doesn't this need to maintain a racecallatomic_ignore path? Without looking too deeply, this seems like it no-ops any atomic operations inside the racedata/racearena structure.
Related details
Attention is currently required from:
- Austin Clements
- Jayanth Krishnamurthy
- Keith Randall
- Martin Möhrmann
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Jayanth Krishnamurthy (Gerrit)
Nov 11, 2025, 5:26:17 PM (2 days ago) Nov 11
to goph...@pubsubhelper.golang.org, Go LUCI, Paul Murphy, Gopher Robot, golang-co...@googlegroups.com
Attention needed from Paul Murphy
Jayanth Krishnamurthy added 4 comments![]( "Open in Gerrit")
File src/runtime/race_ppc64le.s
Line 388, Patchset 8 (Latest):
MOVD addr+0(FP), R3Paul Murphy . unresolved
I'm not sure these are correct? No other implementation does this, and the tsan performs this op.
Jayanth Krishnamurthy
Agree. The __tsan_go_atomic*_compare_exchange hooks already perform the actual CAS operation with full memory ordering. I’ll remove the redundant LL/SC sequence so that the PPC64LE path matches the other implementations and simply calls into TSAN, returning its result directly.
Line 455, Patchset 8 (Latest):
// R6 = addr of incoming arg listPaul Murphy . resolved
Should R7 be mentioned here?
Jayanth Krishnamurthy
Acknowledged
Line 464, Patchset 8 (Latest):
ANDCC $3, R7, R11Paul Murphy . unresolved
This doesn't seem right. If this is the argument pointer, isn't it aligned to the type which isn't necessarily 64b?
Jayanth Krishnamurthy
Right, the fixed 4-byte alignment check in racecallatomic will not be helpful here. I’ll remove the hardcoded ANDCC $3, R7 check and branch, keep only the nil-address fast-exit, and let TSAN handle any required constraints.
Line 496, Patchset 8 (Latest):
race_ret:Paul Murphy . unresolved
Doesn't this need to maintain a racecallatomic_ignore path? Without looking too deeply, this seems like it no-ops any atomic operations inside the racedata/racearena structure.
Jayanth Krishnamurthy
The racecallatomic_ignore logic is already preserved.
We are checking both the heap arena ([racearenastart, racearenaend)) and the race runtime metadata range ([racedatastart, racedataend)), and are skipping TSAN instrumentation if the target address falls within either.
All early exits simply branch to the shared race_ret epilogue instead of a separate ignore label.
Related details
Attention is currently required from:
- Paul Murphy
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Paul Murphy (Gerrit)
Nov 11, 2025, 6:27:53 PM (2 days ago) Nov 11
to Jayanth Krishnamurthy, goph...@pubsubhelper.golang.org, Go LUCI, Gopher Robot, golang-co...@googlegroups.com
Attention needed from Jayanth Krishnamurthy
Paul Murphy added 1 comment![]( "Open in Gerrit")
File src/runtime/race_ppc64le.s
Line 496, Patchset 8 (Latest):
race_ret:Paul Murphy . unresolved
Jayanth KrishnamurthyDoesn't this need to maintain a racecallatomic_ignore path? Without looking too deeply, this seems like it no-ops any atomic operations inside the racedata/racearena structure.
The racecallatomic_ignore logic is already preserved.
We are checking both the heap arena ([racearenastart, racearenaend)) and the race runtime metadata range ([racedatastart, racedataend)), and are skipping TSAN instrumentation if the target address falls within either.
All early exits simply branch to the shared race_ret epilogue instead of a separate ignore label.
Paul Murphy
Where is it preserved? If operating on the metadata, no atomic op is performed at all.
The existing code disables, performs the atomic op, and re-enables the racedetector. All via Go's tsan api.
Related details
Attention is currently required from:
- Jayanth Krishnamurthy
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Reply all
Reply to author
Forward
0 new messages