runtime: use LSX for unaligned 33~64 byte memmove on loong64
For 33~64 byte copies with unaligned dst/src, use four 128-bit LSX
vector load/store pairs instead of eight scalar MOVV pairs. Aligned
copies fall through to the original scalar path with 3 extra
instructions (OR/AND/BEQ); unaligned copies without LSX support fall
through with 5 (adds MOVBU/BEQ for the HasLSX check).
benchstat (Loongson-3C6000, n=10):
geomean 11.93n -> 11.68n (-2.09%)
Unaligned 64B copy ~3.2n -> ~2.1n (-36%)
Aligned 64B copy (+4.75%, alignment probe cost)
The specific performance test results are as follows.
goos: linux
goarch: loong64
pkg: runtime
cpu: Loongson-3C6000/S @ 2200.00MHz
| old.bench | new.bench |
| sec/op | sec/op vs base |
Memmove/0 2.509n ± 1% 2.537n ± 1% +1.12% (p=0.029 n=10)
Memmove/1 3.185n ± 0% 3.185n ± 0% ~ (p=0.914 n=10)
Memmove/2 2.896n ± 1% 2.902n ± 1% ~ (p=0.591 n=10)
Memmove/3 3.183n ± 0% 3.184n ± 0% +0.03% (p=0.011 n=10)
Memmove/4 2.858n ± 1% 2.869n ± 0% ~ (p=0.072 n=10)
Memmove/5 3.638n ± 0% 3.638n ± 0% ~ (p=1.000 n=10) ¹
Memmove/6 3.638n ± 0% 3.638n ± 0% ~ (p=1.000 n=10) ¹
Memmove/7 3.638n ± 0% 3.638n ± 0% ~ (p=1.000 n=10)
Memmove/8 3.183n ± 0% 3.184n ± 0% ~ (p=0.179 n=10)
Memmove/9 3.638n ± 0% 3.638n ± 0% ~ (p=1.000 n=10) ¹
Memmove/10 3.638n ± 0% 3.638n ± 0% ~ (p=1.000 n=10) ¹
Memmove/11 3.638n ± 0% 3.638n ± 0% ~ (p=1.000 n=10) ¹
Memmove/12 3.638n ± 0% 3.638n ± 0% ~ (p=1.000 n=10) ¹
Memmove/13 3.638n ± 0% 3.638n ± 0% ~ (p=1.000 n=10) ¹
Memmove/14 3.638n ± 0% 3.638n ± 0% ~ (p=1.000 n=10) ¹
Memmove/15 3.638n ± 0% 3.638n ± 0% ~ (p=1.000 n=10)
Memmove/16 3.638n ± 0% 3.638n ± 0% ~ (p=1.000 n=10) ¹
Memmove/32 4.093n ± 0% 4.093n ± 0% ~ (p=1.000 n=10) ¹
Memmove/64 4.775n ± 0% 5.002n ± 0% +4.75% (p=0.000 n=10)
Memmove/128 8.186n ± 0% 8.186n ± 0% ~ (p=1.000 n=10) ¹
Memmove/256 7.276n ± 0% 7.276n ± 0% ~ (p=0.087 n=10)
Memmove/512 9.158n ± 0% 9.165n ± 0% ~ (p=0.126 n=10)
Memmove/1024 17.74n ± 0% 17.74n ± 0% ~ (p=1.000 n=10) ¹
Memmove/2048 31.83n ± 0% 31.83n ± 0% ~ (p=1.000 n=10)
Memmove/4096 60.94n ± 0% 60.94n ± 0% ~ (p=1.000 n=10) ¹
MemmoveOverlap/32 4.548n ± 0% 4.548n ± 0% ~ (p=1.000 n=10) ¹
MemmoveOverlap/64 5.457n ± 0% 5.684n ± 0% +4.17% (p=0.000 n=10)
MemmoveOverlap/128 8.641n ± 0% 8.641n ± 0% ~ (p=1.000 n=10) ¹
MemmoveOverlap/256 11.37n ± 0% 11.37n ± 0% ~ (p=1.000 n=10) ¹
MemmoveOverlap/512 18.35n ± 0% 18.35n ± 0% ~ (p=0.276 n=10)
MemmoveOverlap/1024 31.99n ± 0% 31.98n ± 0% ~ (p=0.094 n=10)
MemmoveOverlap/2048 47.52n ± 0% 47.52n ± 0% ~ (p=1.000 n=10) ¹
MemmoveOverlap/4096 97.25n ± 0% 97.27n ± 0% ~ (p=0.052 n=10)
MemmoveUnalignedDst/0 2.776n ± 0% 2.788n ± 1% +0.41% (p=0.004 n=10)
MemmoveUnalignedDst/1 3.638n ± 0% 3.638n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedDst/2 3.217n ± 0% 3.218n ± 0% ~ (p=0.492 n=10)
MemmoveUnalignedDst/3 3.638n ± 0% 3.638n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedDst/4 3.245n ± 0% 3.246n ± 0% ~ (p=0.400 n=10)
MemmoveUnalignedDst/5 4.093n ± 0% 4.093n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedDst/6 4.093n ± 0% 4.093n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedDst/7 4.093n ± 0% 4.093n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedDst/8 3.638n ± 0% 3.638n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedDst/9 4.093n ± 0% 4.093n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedDst/10 4.093n ± 0% 4.093n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedDst/11 4.093n ± 0% 4.093n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedDst/12 4.093n ± 0% 4.093n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedDst/13 4.093n ± 0% 4.093n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedDst/14 4.093n ± 0% 4.093n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedDst/15 4.093n ± 0% 4.093n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedDst/16 4.093n ± 0% 4.093n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedDst/32 4.320n ± 0% 4.320n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedDst/64 7.731n ± 0% 5.457n ± 0% -29.41% (p=0.000 n=10)
MemmoveUnalignedDst/128 11.82n ± 0% 10.91n ± 0% -7.70% (p=0.000 n=10)
MemmoveUnalignedDst/256 10.46n ± 0% 10.46n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedDst/512 12.53n ± 1% 12.53n ± 1% ~ (p=0.869 n=10)
MemmoveUnalignedDst/1024 23.63n ± 0% 23.53n ± 0% -0.44% (p=0.000 n=10)
MemmoveUnalignedDst/2048 47.48n ± 0% 47.65n ± 0% +0.36% (p=0.000 n=10)
MemmoveUnalignedDst/4096 91.04n ± 0% 91.03n ± 0% ~ (p=0.748 n=10)
MemmoveUnalignedDstOverlap/32 6.139n ± 0% 6.139n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedDstOverlap/64 9.323n ± 0% 8.186n ± 0% -12.20% (p=0.000 n=10)
MemmoveUnalignedDstOverlap/128 9.765n ± 0% 9.736n ± 0% -0.30% (p=0.003 n=10)
MemmoveUnalignedDstOverlap/256 12.28n ± 0% 12.28n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedDstOverlap/512 18.20n ± 0% 18.18n ± 0% ~ (p=0.420 n=10)
MemmoveUnalignedDstOverlap/1024 34.10n ± 0% 34.10n ± 0% ~ (p=1.000 n=10)
MemmoveUnalignedDstOverlap/2048 47.75n ± 0% 47.75n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedDstOverlap/4096 98.23n ± 0% 98.23n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrc/0 2.780n ± 0% 2.785n ± 1% ~ (p=0.109 n=10)
MemmoveUnalignedSrc/1 3.638n ± 0% 3.638n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrc/2 3.187n ± 0% 3.187n ± 0% ~ (p=0.568 n=10)
MemmoveUnalignedSrc/3 3.638n ± 0% 3.638n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrc/4 3.188n ± 0% 3.188n ± 0% ~ (p=0.621 n=10)
MemmoveUnalignedSrc/5 4.093n ± 0% 4.093n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrc/6 4.093n ± 0% 4.093n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrc/7 4.093n ± 0% 4.093n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrc/8 3.638n ± 0% 3.638n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrc/9 4.093n ± 0% 4.093n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrc/10 4.093n ± 0% 4.093n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrc/11 4.093n ± 0% 4.093n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrc/12 4.093n ± 0% 4.093n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrc/13 4.093n ± 0% 4.093n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrc/14 4.093n ± 0% 4.093n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrc/15 4.093n ± 0% 4.093n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrc/16 4.093n ± 0% 4.093n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrc/32 4.320n ± 0% 4.320n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrc/64 5.230n ± 0% 5.457n ± 0% +4.34% (p=0.000 n=10)
MemmoveUnalignedSrc/128 8.641n ± 0% 8.641n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrc/256 7.731n ± 0% 7.731n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrc/512 12.63n ± 0% 12.60n ± 0% ~ (p=0.101 n=10)
MemmoveUnalignedSrc/1024 23.50n ± 0% 23.58n ± 0% +0.34% (p=0.003 n=10)
MemmoveUnalignedSrc/2048 45.15n ± 0% 45.11n ± 0% -0.09% (p=0.001 n=10)
MemmoveUnalignedSrc/4096 91.39n ± 1% 91.26n ± 0% -0.15% (p=0.030 n=10)
MemmoveUnalignedSrcDst/f_16_0 4.548n ± 0% 4.548n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrcDst/b_16_0 4.548n ± 0% 4.548n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrcDst/f_16_1 4.548n ± 0% 4.548n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrcDst/b_16_1 4.548n ± 0% 4.548n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrcDst/f_16_4 4.548n ± 0% 4.548n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrcDst/b_16_4 4.548n ± 0% 4.548n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrcDst/f_16_7 4.548n ± 0% 4.548n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrcDst/b_16_7 4.548n ± 0% 4.548n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrcDst/f_64_0 5.912n ± 0% 6.139n ± 0% +3.84% (p=0.000 n=10)
MemmoveUnalignedSrcDst/b_64_0 5.912n ± 0% 6.139n ± 0% +3.84% (p=0.000 n=10)
MemmoveUnalignedSrcDst/f_64_1 9.271n ± 0% 5.912n ± 0% -36.23% (p=0.000 n=10)
MemmoveUnalignedSrcDst/b_64_1 8.188n ± 0% 5.912n ± 0% -27.79% (p=0.000 n=10)
MemmoveUnalignedSrcDst/f_64_4 9.267n ± 0% 5.912n ± 0% -36.20% (p=0.000 n=10)
MemmoveUnalignedSrcDst/b_64_4 8.188n ± 0% 5.912n ± 0% -27.79% (p=0.000 n=10)
MemmoveUnalignedSrcDst/f_64_7 9.264n ± 0% 5.912n ± 0% -36.18% (p=0.000 n=10)
MemmoveUnalignedSrcDst/b_64_7 8.187n ± 0% 5.912n ± 0% -27.79% (p=0.000 n=10)
MemmoveUnalignedSrcDst/f_256_0 8.186n ± 0% 8.186n ± 0% ~ (p=0.474 n=10)
MemmoveUnalignedSrcDst/b_256_0 11.66n ± 0% 11.65n ± 0% ~ (p=0.174 n=10)
MemmoveUnalignedSrcDst/f_256_1 10.91n ± 0% 10.91n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrcDst/b_256_1 11.82n ± 0% 11.82n ± 0% ~ (p=0.211 n=10)
MemmoveUnalignedSrcDst/f_256_4 11.14n ± 0% 11.14n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrcDst/b_256_4 11.82n ± 0% 11.82n ± 0% ~ (p=1.000 n=10)
MemmoveUnalignedSrcDst/f_256_7 11.79n ± 0% 11.79n ± 0% ~ (p=1.000 n=10)
MemmoveUnalignedSrcDst/b_256_7 11.82n ± 0% 11.82n ± 0% ~ (p=1.000 n=10)
MemmoveUnalignedSrcDst/f_4096_0 92.32n ± 0% 92.32n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrcDst/b_4096_0 90.27n ± 0% 90.27n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrcDst/f_4096_1 93.70n ± 0% 93.52n ± 0% -0.19% (p=0.000 n=10)
MemmoveUnalignedSrcDst/b_4096_1 91.38n ± 0% 91.41n ± 0% ~ (p=0.444 n=10)
MemmoveUnalignedSrcDst/f_4096_4 92.55n ± 0% 92.53n ± 0% ~ (p=0.382 n=10)
MemmoveUnalignedSrcDst/b_4096_4 91.35n ± 0% 91.38n ± 0% +0.04% (p=0.022 n=10)
MemmoveUnalignedSrcDst/f_4096_7 90.95n ± 0% 90.95n ± 0% ~ (p=1.000 n=10)
MemmoveUnalignedSrcDst/b_4096_7 91.37n ± 0% 91.39n ± 0% +0.03% (p=0.011 n=10)
MemmoveUnalignedSrcDst/f_65536_0 2.573µ ± 0% 2.573µ ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrcDst/b_65536_0 2.573µ ± 0% 2.573µ ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrcDst/f_65536_1 2.573µ ± 0% 2.573µ ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrcDst/b_65536_1 2.574µ ± 0% 2.574µ ± 0% ~ (p=0.474 n=10)
MemmoveUnalignedSrcDst/f_65536_4 2.573µ ± 0% 2.573µ ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrcDst/b_65536_4 2.574µ ± 0% 2.574µ ± 0% ~ (p=0.474 n=10)
MemmoveUnalignedSrcDst/f_65536_7 2.574µ ± 0% 2.574µ ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrcDst/b_65536_7 2.574µ ± 0% 2.574µ ± 0% ~ (p=1.000 n=10)
MemmoveUnalignedSrcOverlap/32 7.731n ± 0% 7.731n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrcOverlap/64 11.370n ± 0% 8.415n ± 0% -25.99% (p=0.000 n=10)
MemmoveUnalignedSrcOverlap/128 12.51n ± 0% 11.13n ± 0% -11.00% (p=0.000 n=10)
MemmoveUnalignedSrcOverlap/256 11.83n ± 0% 11.83n ± 0% ~ (p=1.000 n=10)
MemmoveUnalignedSrcOverlap/512 18.99n ± 0% 18.99n ± 0% ~ (p=1.000 n=10) ¹
MemmoveUnalignedSrcOverlap/1024 33.76n ± 0% 33.76n ± 0% ~ (p=0.864 n=10)
MemmoveUnalignedSrcOverlap/2048 45.97n ± 0% 45.81n ± 0% -0.35% (p=0.000 n=10)
MemmoveUnalignedSrcOverlap/4096 97.24n ± 0% 96.70n ± 0% -0.56% (p=0.000 n=10)
MemmoveKnownSize112 4.623n ± 0% 4.623n ± 0% ~ (p=0.211 n=10)
MemmoveKnownSize128 3.867n ± 0% 3.867n ± 0% ~ (p=0.340 n=10)
MemmoveKnownSize192 5.685n ± 0% 5.684n ± 0% ~ (p=0.059 n=10)
MemmoveKnownSize248 7.882n ± 0% 7.882n ± 0% ~ (p=0.211 n=10)
MemmoveKnownSize256 7.505n ± 0% 7.504n ± 0% -0.02% (p=0.007 n=10)
MemmoveKnownSize512 14.78n ± 0% 14.78n ± 0% ~ (p=1.000 n=10) ¹
MemmoveKnownSize1024 29.34n ± 0% 29.33n ± 0% ~ (p=0.625 n=10)
geomean 11.93n 11.68n -2.09%
¹ all samples are equal
diff --git a/src/runtime/memmove_loong64.s b/src/runtime/memmove_loong64.s
index 0d0af68..ec97b1a 100644
--- a/src/runtime/memmove_loong64.s
+++ b/src/runtime/memmove_loong64.s
@@ -103,9 +103,10 @@
SGTU $33, R6, R9
BNE R9, move_17through32
- // >= 33 bytes and < 65 bytes
+ // >= 33 bytes and < 65 bytes: dispatch to scalar or vector
+ // (LSX) handling depending on dst/src alignment.
SGTU $65, R6, R9
- BNE R9, move_33through64
+ BNE R9, move_33through64_dispatch
// >= 65 bytes and < 256 bytes
SGTU $256, R6, R9
@@ -180,6 +181,33 @@
MOVV R18, -8(R7)
RET
+// move_33through64_dispatch only runs for the 33~64 byte bucket.
+// Aligned dst/src (the common case) falls straight through to the
+// original scalar move_33through64 after 3 extra instructions
+// (OR/AND/BEQ). Unaligned dst/src without LSX support falls through
+// after 5 extra instructions (OR/AND/BEQ/MOVBU/BEQ). Unaligned
+// dst/src with LSX available uses four 128-bit vector load/store
+// pairs instead of the eight scalar MOVV load/store pairs used by
+// move_33through64.
+move_33through64_dispatch:
+ OR R4, R5, R9
+ AND $7, R9
+ BEQ R9, move_33through64
+ MOVBU internal∕cpu·Loong64+const_offsetLOONG64HasLSX(SB), R9
+ BEQ R9, move_33through64
+ JMP lsx_move_33through64
+
+lsx_move_33through64:
+ VMOVQ (R5), V0
+ VMOVQ 16(R5), V1
+ VMOVQ -32(R8), V2
+ VMOVQ -16(R8), V3
+ VMOVQ V0, (R4)
+ VMOVQ V1, 16(R4)
+ VMOVQ V2, -32(R7)
+ VMOVQ V3, -16(R7)
+ RET
+
move_large:
// if (dst > src) && (dst < (src + count))
// regarded as memory overlap
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
| Code-Review | +2 |
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
| Code-Review | +1 |
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |