[go] crypto/sha: implement SHA1 & SHA256 acceleration using Intel SHA extensions
399 views
Skip to first unread message
Gerrit Bot (Gerrit)
Oct 1, 2021, 8:15:26 AM10/1/21
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Gerrit Bot has uploaded this change for review.
crypto/sha: implement SHA1 & SHA256 acceleration using Intel SHA extensions
This change implements SHA1 & SHA256 acceleration using the Intel SHA
extensions if those instructions are enabled. First the internal/cpu
package needs support to detect the CPU extension flag for SHA extensions.
The crypto/sha256 package contains a small refactor so that it matches how
crypto/sha1 & crypto/sha512 are set up and it makes it easier to use
consistent code in how it's decided which implementation can be used based
on CPU support.
All the work here is based on the Intel reference documentation at
https://software.intel.com/content/www/us/en/develop/articles/intel-sha-extensions.html.
Using only the reference documentation avoids issue like in #27443 where
code from another project was converted to Go assembly.
Benchmarks show a close to 2x performance improvement on an AMD Ryzen 5
3600 for SHA1:
name old time/op new time/op delta
Sha/SHA1______16_bytes-12 172ns ± 3% 105ns ± 4% -39.06% (p=0.000 n=20+21)
Sha/SHA1______64_bytes-12 261ns ± 2% 139ns ± 3% -46.85% (p=0.000 n=21+21)
Sha/SHA1_____256_bytes-12 492ns ± 2% 229ns ± 2% -53.41% (p=0.000 n=20+20)
Sha/SHA1______1k_bytes-12 1.17µs ± 1% 0.59µs ± 1% -49.43% (p=0.000 n=20+19)
Sha/SHA1______8k_bytes-12 7.48µs ± 1% 3.94µs ± 1% -47.35% (p=0.000 n=20+21)
Sha/SHA1____256k_bytes-12 232µs ± 2% 122µs ± 1% -47.25% (p=0.000 n=21+20)
Sha/SHA1___1024k_bytes-12 928µs ± 2% 491µs ± 1% -47.12% (p=0.000 n=21+21)
Benchmarks show a close to 4x performance improvement on an AMD Ryzen 5
3600, especially on larger inputs. Even on the smallest it's at least 2x
faster.
name old time/op new time/op delta
Sha/SHA256____16_bytes-12 248ns ± 3% 117ns ± 3% -52.84% (p=0.000 n=20+19)
Sha/SHA256____64_bytes-12 384ns ± 2% 153ns ± 3% -60.10% (p=0.000 n=20+17)
Sha/SHA256___256_bytes-12 786ns ± 1% 249ns ± 3% -68.29% (p=0.000 n=19+19)
Sha/SHA256____1k_bytes-12 2.36µs ± 1% 0.64µs ± 3% -72.93% (p=0.000 n=19+20)
Sha/SHA256____8k_bytes-12 17.0µs ± 2% 4.2µs ± 1% -75.16% (p=0.000 n=20+20)
Sha/SHA256__256k_bytes-12 537µs ± 1% 131µs ± 1% -75.60% (p=0.000 n=20+20)
Sha/SHA256_1024k_bytes-12 2.15ms ± 1% 0.52ms ± 1% -75.60% (p=0.000 n=20+20)
The discussion in #27443 mentions that for SHA1 including this was
debatable, since the algorithm itself is no longer considered safe. I think
that a 2x performance improvement is still significant though and there's
still a lot of places where SHA1 is still used (for example for Git itself).
Of course the SHA1 change can be backed out if this change is only desired
for SHA256 because of that.
Fixes #27443
Change-Id: Ided78cb6533f412747cf4d8d64f10154f16094f9
GitHub-Last-Rev: ff02c85da6570a1a099d6c3d0293ffe99181b202
GitHub-Pull-Request: golang/go#48720
---
M src/crypto/sha256/sha256block_amd64.go
M src/crypto/sha1/sha1block_amd64.go
M src/crypto/sha256/sha256block_amd64.s
M src/internal/cpu/cpu_x86.go
M src/internal/cpu/cpu.go
M src/crypto/sha1/sha1block_amd64.s
M src/crypto/sha256/sha256block_decl.go
7 files changed, 604 insertions(+), 8 deletions(-)
diff --git a/src/crypto/sha1/sha1block_amd64.go b/src/crypto/sha1/sha1block_amd64.go
index 039813d..d32b890 100644
--- a/src/crypto/sha1/sha1block_amd64.go
+++ b/src/crypto/sha1/sha1block_amd64.go
@@ -12,10 +12,16 @@
//go:noescape
func blockAMD64(dig *digest, p []byte)
+//go:noescape
+func blockSHA(dig *digest, p []byte)
+
var useAVX2 = cpu.X86.HasAVX2 && cpu.X86.HasBMI1 && cpu.X86.HasBMI2
+var useSHA = cpu.X86.HasSHA
func block(dig *digest, p []byte) {
- if useAVX2 && len(p) >= 256 {
+ if useSHA {
+ blockSHA(dig, p)
+ } else if useAVX2 && len(p) >= 256 {
// blockAVX2 calculates sha1 for 2 block per iteration
// it also interleaves precalculation for next block.
// So it may read up-to 192 bytes past end of p
diff --git a/src/crypto/sha1/sha1block_amd64.s b/src/crypto/sha1/sha1block_amd64.s
index 42f03fb..4adb2fc 100644
--- a/src/crypto/sha1/sha1block_amd64.s
+++ b/src/crypto/sha1/sha1block_amd64.s
@@ -1498,3 +1498,237 @@
DATA BSWAP_SHUFB_CTL<>+0x18(SB)/4,$0x08090a0b
DATA BSWAP_SHUFB_CTL<>+0x1c(SB)/4,$0x0c0d0e0f
GLOBL BSWAP_SHUFB_CTL<>(SB),RODATA,$32
+
+// SHA1 implementation using the SHA extension. Implemented using the
+// reference implementation found at:
+// https://software.intel.com/content/www/us/en/develop/articles/intel-sha-extensions.html
+
+// Setup register aliases to easier follow the algorithm.
+#define ABCD X0
+#define E0 X1
+#define E1 X2
+#define SHUF_MASK X3
+#define MSG0 X4
+#define MSG1 X5
+#define MSG2 X6
+#define MSG3 X7
+#define ABCD_SAVE X8
+#define E_SAVE X9
+
+TEXT ·blockSHA(SB),NOSPLIT,$0-32
+ MOVQ dig+0(FP), CX
+ MOVQ p_base+8(FP), SI
+ MOVQ p_len+16(FP), DX
+ SHRQ $6, DX
+ SHLQ $6, DX
+ LEAQ (SI)(DX*1), DI
+
+ // Load first 128 bits into xmm register.
+ MOVOU (CX), ABCD
+
+ // Byte shuffle for correct 32 bit dword order. The order
+ // the algorithm expects is that the 32 bit value A lives
+ // in the lowest 32 bits & D in the highest. This inverse
+ // from how they are laid out in memory so we need to reverse
+ // the order of each 32 bit word.
+ PSHUFD $0x1b, ABCD, ABCD
+
+ // Zero out E0 since we only set 32 bits of it and the rest
+ // needs to be zero. E0 needs to be set in the highest 32
+ // bites of the XMM register.
+ PXOR E0, E0
+ PINSRD $3, 16(CX), E0
+
+ // Load the shuffle mask. This is separate from the AVX2 form
+ // since we need this in a different 32 bit order and this
+ // avoid having to use a PSHUFD here which would be needed
+ // if BSWAP_SHUFB_CTL was reused here.
+ MOVO BSWAP_SHUF_MASK<>(SB), SHUF_MASK
+
+ // Skip if we accidentally have a zero sized block.
+ CMPQ SI, DI
+ JEQ end
+
+loop:
+
+ // Save working variables.
+ MOVO ABCD, ABCD_SAVE
+ MOVO E0, E_SAVE
+
+ // Rounds 0 - 3.
+ MOVOU (SI), MSG0
+ PSHUFB SHUF_MASK, MSG0
+ PADDD MSG0, E0
+ MOVO ABCD, E1
+ SHA1RNDS4 $0, E0, ABCD
+
+ // Rounds 4 - 7.
+ MOVOU 16(SI), MSG1
+ PSHUFB SHUF_MASK, MSG1
+ SHA1NEXTE MSG1, E1
+ MOVOA ABCD, E0
+ SHA1RNDS4 $0, E1, ABCD
+ SHA1MSG1 MSG1, MSG0
+
+ // Rounds 8 - 11.
+ MOVOU 32(SI), MSG2
+ PSHUFB SHUF_MASK, MSG2
+ SHA1NEXTE MSG2, E0
+ MOVOA ABCD, E1
+ SHA1RNDS4 $0, E0, ABCD
+ SHA1MSG1 MSG2, MSG1
+ PXOR MSG2, MSG0
+
+ // Rounds 12 - 15.
+ MOVOU 48(SI), MSG3
+ PSHUFB SHUF_MASK, MSG3
+ SHA1NEXTE MSG3, E1
+ MOVOA ABCD, E0
+ SHA1MSG2 MSG3, MSG0
+ SHA1RNDS4 $0, E1, ABCD
+ SHA1MSG1 MSG3, MSG2
+ PXOR MSG3, MSG1
+
+ // Rounds 16 - 19.
+ SHA1NEXTE MSG0, E0
+ MOVOA ABCD, E1
+ SHA1MSG2 MSG0, MSG1
+ SHA1RNDS4 $0, E0, ABCD
+ SHA1MSG1 MSG0, MSG3
+ PXOR MSG0, MSG2
+
+ // Rounds 20 - 23.
+ SHA1NEXTE MSG1, E1
+ MOVOA ABCD, E0
+ SHA1MSG2 MSG1, MSG2
+ SHA1RNDS4 $1, E1, ABCD
+ SHA1MSG1 MSG1, MSG0
+ PXOR MSG1, MSG3
+
+ // Rounds 24 - 27.
+ SHA1NEXTE MSG2, E0
+ MOVOA ABCD, E1
+ SHA1MSG2 MSG2, MSG3
+ SHA1RNDS4 $1, E0, ABCD
+ SHA1MSG1 MSG2, MSG1
+ PXOR MSG2, MSG0
+
+ // Rounds 28 - 31.
+ SHA1NEXTE MSG3, E1
+ MOVOA ABCD, E0
+ SHA1MSG2 MSG3, MSG0
+ SHA1RNDS4 $1, E1, ABCD
+ SHA1MSG1 MSG3, MSG2
+ PXOR MSG3, MSG1
+
+ // Rounds 32 - 35.
+ SHA1NEXTE MSG0, E0
+ MOVOA ABCD, E1
+ SHA1MSG2 MSG0, MSG1
+ SHA1RNDS4 $1, E0, ABCD
+ SHA1MSG1 MSG0, MSG3
+ PXOR MSG0, MSG2
+
+ // Rounds 36 - 39.
+ SHA1NEXTE MSG1, E1
+ MOVOA ABCD, E0
+ SHA1MSG2 MSG1, MSG2
+ SHA1RNDS4 $1, E1, ABCD
+ SHA1MSG1 MSG1, MSG0
+ PXOR MSG1, MSG3
+
+ // Rounds 40 - 43.
+ SHA1NEXTE MSG2, E0
+ MOVOA ABCD, E1
+ SHA1MSG2 MSG2, MSG3
+ SHA1RNDS4 $2, E0, ABCD
+ SHA1MSG1 MSG2, MSG1
+ PXOR MSG2, MSG0
+
+ // Rounds 44 - 47.
+ SHA1NEXTE MSG3, E1
+ MOVOA ABCD, E0
+ SHA1MSG2 MSG3, MSG0
+ SHA1RNDS4 $2, E1, ABCD
+ SHA1MSG1 MSG3, MSG2
+ PXOR MSG3, MSG1
+
+ // Rounds 48 - 51.
+ SHA1NEXTE MSG0, E0
+ MOVOA ABCD, E1
+ SHA1MSG2 MSG0, MSG1
+ SHA1RNDS4 $2, E0, ABCD
+ SHA1MSG1 MSG0, MSG3
+ PXOR MSG0, MSG2
+
+ // Rounds 52 - 55.
+ SHA1NEXTE MSG1, E1
+ MOVOA ABCD, E0
+ SHA1MSG2 MSG1, MSG2
+ SHA1RNDS4 $2, E1, ABCD
+ SHA1MSG1 MSG1, MSG0
+ PXOR MSG1, MSG3
+
+ // Rounds 56 - 59.
+ SHA1NEXTE MSG2, E0
+ MOVO ABCD, E1
+ SHA1MSG2 MSG2, MSG3
+ SHA1RNDS4 $2, E0, ABCD
+ SHA1MSG1 MSG2, MSG1
+ PXOR MSG2, MSG0
+
+ // Rounds 60 - 63.
+ SHA1NEXTE MSG3, E1
+ MOVO ABCD, E0
+ SHA1MSG2 MSG3, MSG0
+ SHA1RNDS4 $3, E1, ABCD
+ SHA1MSG1 MSG3, MSG2
+ PXOR MSG3, MSG1
+
+ // Rounds 64 - 67.
+ SHA1NEXTE MSG0, E0
+ MOVO ABCD, E1
+ SHA1MSG2 MSG0, MSG1
+ SHA1RNDS4 $3, E0, ABCD
+ SHA1MSG1 MSG0, MSG3
+ PXOR MSG0, MSG2
+
+ // Rounds 68 - 71.
+ SHA1NEXTE MSG1, E1
+ MOVO ABCD, E0
+ SHA1MSG2 MSG1, MSG2
+ SHA1RNDS4 $3, E1, ABCD
+ PXOR MSG1, MSG3
+
+ // Rounds 72 - 75.
+ SHA1NEXTE MSG2, E0
+ MOVO ABCD, E1
+ SHA1MSG2 MSG2, MSG3
+ SHA1RNDS4 $3, E0, ABCD
+
+ // Rounds 76 - 79.
+ SHA1NEXTE MSG3, E1
+ MOVO ABCD, E0
+ SHA1RNDS4 $3, E1, ABCD
+
+ // Complete block.
+ SHA1NEXTE E_SAVE, E0
+ PADDD ABCD_SAVE, ABCD
+
+ // Check if we need to process another block.
+ ADDQ $64, SI
+ CMPQ SI, DI
+ JB loop
+
+ // Update digest.
+ PSHUFD $0x1b, ABCD, ABCD
+ MOVOU ABCD, (CX)
+ PEXTRD $3, E0, 16(CX)
+end:
+ RET
+
+DATA BSWAP_SHUF_MASK<>+0x00(SB)/4,$0x0c0d0e0f
+DATA BSWAP_SHUF_MASK<>+0x04(SB)/4,$0x08090a0b
+DATA BSWAP_SHUF_MASK<>+0x08(SB)/4,$0x04050607
+DATA BSWAP_SHUF_MASK<>+0x0c(SB)/4,$0x00010203
+GLOBL BSWAP_SHUF_MASK<>(SB),RODATA,$16
diff --git a/src/crypto/sha256/sha256block_amd64.go b/src/crypto/sha256/sha256block_amd64.go
index 27464e2..ed3d7a3 100644
--- a/src/crypto/sha256/sha256block_amd64.go
+++ b/src/crypto/sha256/sha256block_amd64.go
@@ -6,4 +6,24 @@
import "internal/cpu"
+//go:noescape
+func blockAVX2(dig *digest, p []byte)
+
+//go:noescape
+func blockAMD64(dig *digest, p []byte)
+
+//go:noescape
+func blockSHA(dig *digest, p []byte)
+
var useAVX2 = cpu.X86.HasAVX2 && cpu.X86.HasBMI2
+var useSHA = cpu.X86.HasSHA
+
+func block(dig *digest, p []byte) {
+ if useSHA {
+ blockSHA(dig, p)
+ } else if useAVX2 {
+ blockAVX2(dig, p)
+ } else {
+ blockAMD64(dig, p)
+ }
+}
diff --git a/src/crypto/sha256/sha256block_amd64.s b/src/crypto/sha256/sha256block_amd64.s
index f6af47c..83d4e94 100644
--- a/src/crypto/sha256/sha256block_amd64.s
+++ b/src/crypto/sha256/sha256block_amd64.s
@@ -550,10 +550,7 @@
; \
ADDL y3, h // h = t1 + S0 + MAJ // --
-TEXT ·block(SB), 0, $536-32
- CMPB ·useAVX2(SB), $1
- JE avx2
-
+TEXT ·blockAMD64(SB), 0, $536-32
MOVQ p_base+8(FP), SI
MOVQ p_len+16(FP), DX
SHRQ $6, DX
@@ -668,7 +665,7 @@
end:
RET
-avx2:
+TEXT ·blockAVX2(SB), 0, $536-32
MOVQ dig+0(FP), CTX // d.h[8]
MOVQ p_base+8(FP), INP
MOVQ p_len+16(FP), NUM_BYTES
@@ -1029,3 +1026,279 @@
DATA K256<>+0x1fc(SB)/4, $0xc67178f2
GLOBL K256<>(SB), (NOPTR + RODATA), $512
+
+// SHA256 implementation using the SHA extension. Implemented using the
+// reference implementation found at:
+// https://software.intel.com/content/www/us/en/develop/articles/intel-sha-extensions.html
+
+// Setup register aliases to easier follow the algorithm.
+#define MSG X0
+#define STATE0 X1
+#define STATE1 X2
+#define MSGTMP0 X3
+#define MSGTMP1 X4
+#define MSGTMP2 X5
+#define MSGTMP3 X6
+#define MSGTMP4 X7
+#define SHUF_MASK X8
+#define ABEF_SAVE X9
+#define CDGH_SAVE X10
+#define SHA256CONSTANTS BX
+
+TEXT ·blockSHA(SB), NOSPLIT, $0-32
+
+ MOVQ dig+0(FP), CX
+ MOVQ p_base+8(FP), SI
+ MOVQ p_len+16(FP), DX
+ SHRQ $6, DX
+ SHLQ $6, DX
+ LEAQ (SI)(DX*1), DI
+
+ MOVOU (CX), STATE0
+ MOVOU 16(CX), STATE1
+
+ // Byte shuffle for correct 32 bit dword order.
+ // The algorithm assumes 8 32 bit working variables called
+ // A through H. CDGH need to be stored in one XMM register &
+ // ABEF in another one. So the desired end state here is ABEF
+ // (each a 32 bit dword in that order from high to low) in
+ // STATE0 & CDGH in STATE1.
+ //
+ // We start off with DCBA in STATE0 & HGFE in STATE1
+ //
+ // First, shuffle DCBA -> CDAB
+ PSHUFD $0xb1, STATE0, STATE0
+ // Shuffle HGFE -> EFGH
+ PSHUFD $0x1b, STATE1, STATE1
+ // Copy EFGH into temporary register since blending would
+ // otherwise lose data.
+ MOVO STATE1, MSGTMP0
+ // Blend CDAB into EFGH to result in CDGH
+ PBLENDW $0xf0, STATE0, STATE1
+ // Shift AB (8 bytes) from CDAB into EFGH to result in ABEF.
+ PALIGNR $8, MSGTMP0, STATE0
+
+ MOVO flip_mask<>(SB), SHUF_MASK
+
+ // Reuses the existing constant table, but it means that each
+ // time SHA256CONSTANTS is used the offset is doubled since
+ // K256 contains duplicate entries for the AVX2 path.
+ LEAQ K256<>(SB), SHA256CONSTANTS
+
+ // Skip if we accidentally have a zero sized block.
+ CMPQ SI, DI
+ JEQ end
+
+loop:
+
+ // Save working variables.
+ MOVO STATE0, ABEF_SAVE
+ MOVO STATE1, CDGH_SAVE
+
+ // Rounds 0-3.
+ MOVOU (SI), MSG
+ PSHUFB SHUF_MASK, MSG
+ MOVO MSG, MSGTMP0
+ PADDD 0*16(SHA256CONSTANTS), MSG
+ SHA256RNDS2 MSG, STATE0, STATE1
+ PSHUFD $0x0e, MSG, MSG
+ SHA256RNDS2 MSG, STATE1, STATE0
+
+ // Rounds 4-7.
+ MOVOU 16(SI), MSG
+ PSHUFB SHUF_MASK, MSG
+ MOVO MSG, MSGTMP1
+ PADDD 2*16(SHA256CONSTANTS), MSG
+ SHA256RNDS2 MSG, STATE0, STATE1
+ PSHUFD $0x0e, MSG, MSG
+ SHA256RNDS2 MSG, STATE1, STATE0
+ SHA256MSG1 MSGTMP1, MSGTMP0
+
+ // Rounds 8-11.
+ MOVOU 32(SI), MSG
+ PSHUFB SHUF_MASK, MSG
+ MOVO MSG, MSGTMP2
+ PADDD 4*16(SHA256CONSTANTS), MSG
+ SHA256RNDS2 MSG, STATE0, STATE1
+ PSHUFD $0x0e, MSG, MSG
+ SHA256RNDS2 MSG, STATE1, STATE0
+ SHA256MSG1 MSGTMP2, MSGTMP1
+
+ // Rounds 12-15.
+ MOVOU 48(SI), MSG
+ PSHUFB SHUF_MASK, MSG
+ MOVO MSG, MSGTMP3
+ PADDD 6*16(SHA256CONSTANTS), MSG
+ SHA256RNDS2 MSG, STATE0, STATE1
+ MOVO MSGTMP3, MSGTMP4
+ PALIGNR $4, MSGTMP2, MSGTMP4
+ PADDD MSGTMP4, MSGTMP0
+ SHA256MSG2 MSGTMP3, MSGTMP0
+ PSHUFD $0x0e, MSG, MSG
+ SHA256RNDS2 MSG, STATE1, STATE0
+ SHA256MSG1 MSGTMP3, MSGTMP2
+
+ // Rounds 16-19.
+ MOVO MSGTMP0, MSG
+ PADDD 8*16(SHA256CONSTANTS), MSG
+ SHA256RNDS2 MSG, STATE0, STATE1
+ MOVO MSGTMP0, MSGTMP4
+ PALIGNR $4, MSGTMP3, MSGTMP4
+ PADDD MSGTMP4, MSGTMP1
+ SHA256MSG2 MSGTMP0, MSGTMP1
+ PSHUFD $0x0e, MSG, MSG
+ SHA256RNDS2 MSG, STATE1, STATE0
+ SHA256MSG1 MSGTMP0, MSGTMP3
+
+ // Rounds 20-23.
+ MOVO MSGTMP1, MSG
+ PADDD 10*16(SHA256CONSTANTS), MSG
+ SHA256RNDS2 MSG, STATE0, STATE1
+ MOVO MSGTMP1, MSGTMP4
+ PALIGNR $4, MSGTMP0, MSGTMP4
+ PADDD MSGTMP4, MSGTMP2
+ SHA256MSG2 MSGTMP1, MSGTMP2
+ PSHUFD $0x0e, MSG, MSG
+ SHA256RNDS2 MSG, STATE1, STATE0
+ SHA256MSG1 MSGTMP1, MSGTMP0
+
+ // Rounds 24-27.
+ MOVO MSGTMP2, MSG
+ PADDD 12*16(SHA256CONSTANTS), MSG
+ SHA256RNDS2 MSG, STATE0, STATE1
+ MOVO MSGTMP2, MSGTMP4
+ PALIGNR $4, MSGTMP1, MSGTMP4
+ PADDD MSGTMP4, MSGTMP3
+ SHA256MSG2 MSGTMP2, MSGTMP3
+ PSHUFD $0x0e, MSG, MSG
+ SHA256RNDS2 MSG, STATE1, STATE0
+ SHA256MSG1 MSGTMP2, MSGTMP1
+
+ // Rounds 28-31.
+ MOVO MSGTMP3, MSG
+ PADDD 14*16(SHA256CONSTANTS), MSG
+ SHA256RNDS2 MSG, STATE0, STATE1
+ MOVO MSGTMP3, MSGTMP4
+ PALIGNR $4, MSGTMP2, MSGTMP4
+ PADDD MSGTMP4, MSGTMP0
+ SHA256MSG2 MSGTMP3, MSGTMP0
+ PSHUFD $0x0e, MSG, MSG
+ SHA256RNDS2 MSG, STATE1, STATE0
+ SHA256MSG1 MSGTMP3, MSGTMP2
+
+ // Rounds 32-35.
+ MOVO MSGTMP0, MSG
+ PADDD 16*16(SHA256CONSTANTS), MSG
+ SHA256RNDS2 MSG, STATE0, STATE1
+ MOVO MSGTMP0, MSGTMP4
+ PALIGNR $4, MSGTMP3, MSGTMP4
+ PADDD MSGTMP4, MSGTMP1
+ SHA256MSG2 MSGTMP0, MSGTMP1
+ PSHUFD $0x0e, MSG, MSG
+ SHA256RNDS2 MSG, STATE1, STATE0
+ SHA256MSG1 MSGTMP0, MSGTMP3
+
+ // Rounds 36-39.
+ MOVO MSGTMP1, MSG
+ PADDD 18*16(SHA256CONSTANTS), MSG
+ SHA256RNDS2 MSG, STATE0, STATE1
+ MOVO MSGTMP1, MSGTMP4
+ PALIGNR $4, MSGTMP0, MSGTMP4
+ PADDD MSGTMP4, MSGTMP2
+ SHA256MSG2 MSGTMP1, MSGTMP2
+ PSHUFD $0x0e, MSG, MSG
+ SHA256RNDS2 MSG, STATE1, STATE0
+ SHA256MSG1 MSGTMP1, MSGTMP0
+
+ // Rounds 40-43.
+ MOVO MSGTMP2, MSG
+ PADDD 20*16(SHA256CONSTANTS), MSG
+ SHA256RNDS2 MSG, STATE0, STATE1
+ MOVO MSGTMP2, MSGTMP4
+ PALIGNR $4, MSGTMP1, MSGTMP4
+ PADDD MSGTMP4, MSGTMP3
+ SHA256MSG2 MSGTMP2, MSGTMP3
+ PSHUFD $0x0e, MSG, MSG
+ SHA256RNDS2 MSG, STATE1, STATE0
+ SHA256MSG1 MSGTMP2, MSGTMP1
+
+ // Rounds 44-47.
+ MOVO MSGTMP3, MSG
+ PADDD 22*16(SHA256CONSTANTS), MSG
+ SHA256RNDS2 MSG, STATE0, STATE1
+ MOVO MSGTMP3, MSGTMP4
+ PALIGNR $4, MSGTMP2, MSGTMP4
+ PADDD MSGTMP4, MSGTMP0
+ SHA256MSG2 MSGTMP3, MSGTMP0
+ PSHUFD $0x0e, MSG, MSG
+ SHA256RNDS2 MSG, STATE1, STATE0
+ SHA256MSG1 MSGTMP3, MSGTMP2
+
+ // Rounds 48-51.
+ MOVO MSGTMP0, MSG
+ PADDD 24*16(SHA256CONSTANTS), MSG
+ SHA256RNDS2 MSG, STATE0, STATE1
+ MOVO MSGTMP0, MSGTMP4
+ PALIGNR $4, MSGTMP3, MSGTMP4
+ PADDD MSGTMP4, MSGTMP1
+ SHA256MSG2 MSGTMP0, MSGTMP1
+ PSHUFD $0x0e, MSG, MSG
+ SHA256RNDS2 MSG, STATE1, STATE0
+ SHA256MSG1 MSGTMP0, MSGTMP3
+
+ // Rounds 52-55.
+ MOVO MSGTMP1, MSG
+ PADDD 26*16(SHA256CONSTANTS), MSG
+ SHA256RNDS2 MSG, STATE0, STATE1
+ MOVO MSGTMP1, MSGTMP4
+ PALIGNR $4, MSGTMP0, MSGTMP4
+ PADDD MSGTMP4, MSGTMP2
+ SHA256MSG2 MSGTMP1, MSGTMP2
+ PSHUFD $0x0e, MSG, MSG
+ SHA256RNDS2 MSG, STATE1, STATE0
+
+ // Rounds 56-59.
+ MOVO MSGTMP2, MSG
+ PADDD 28*16(SHA256CONSTANTS), MSG
+ SHA256RNDS2 MSG, STATE0, STATE1
+ MOVO MSGTMP2, MSGTMP4
+ PALIGNR $4, MSGTMP1, MSGTMP4
+ PADDD MSGTMP4, MSGTMP3
+ SHA256MSG2 MSGTMP2, MSGTMP3
+ PSHUFD $0x0e, MSG, MSG
+ SHA256RNDS2 MSG, STATE1, STATE0
+
+ // Rounds 60-63.
+ MOVO MSGTMP3, MSG
+ PADDD 30*16(SHA256CONSTANTS), MSG
+ SHA256RNDS2 MSG, STATE0, STATE1
+ PSHUFD $0x0e, MSG, MSG
+ SHA256RNDS2 MSG, STATE1, STATE0
+
+ // Mix in previously saved values.
+ PADDD ABEF_SAVE, STATE0
+ PADDD CDGH_SAVE, STATE1
+
+ // Check if we need to process another block.
+ ADDQ $64, SI
+ CMPQ SI, DI
+ JB loop
+
+ // Write hash values back in the correct order. This is the
+ // inverse of what was done in the setup.
+ // Shuffle ABEF -> FEBA
+ PSHUFD $0x1b, STATE0, STATE0
+ // Shuffle CDGH -> DCHG
+ PSHUFD $0xb1, STATE1, STATE1
+ MOVO STATE0, MSGTMP0
+
+ // Blend DCGH & FEBA to result in DCBA
+ PBLENDW $0xf0, STATE1, STATE0
+ // Shift FEBA into DCGH to result in GHFE
+ PALIGNR $8, MSGTMP0, STATE1
+
+ // Update digest.
+ MOVOU STATE0, (CX)
+ MOVOU STATE1, 16(CX)
+end:
+ RET
diff --git a/src/crypto/sha256/sha256block_decl.go b/src/crypto/sha256/sha256block_decl.go
index a6bb396..e89fbaa 100644
--- a/src/crypto/sha256/sha256block_decl.go
+++ b/src/crypto/sha256/sha256block_decl.go
@@ -2,8 +2,8 @@
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
-//go:build 386 || amd64 || s390x || ppc64le
-// +build 386 amd64 s390x ppc64le
+//go:build 386 || s390x || ppc64le
+// +build 386 s390x ppc64le
package sha256
diff --git a/src/internal/cpu/cpu.go b/src/internal/cpu/cpu.go
index 4f0c5d2..bf0630e 100644
--- a/src/internal/cpu/cpu.go
+++ b/src/internal/cpu/cpu.go
@@ -41,6 +41,7 @@
HasSSSE3 bool
HasSSE41 bool
HasSSE42 bool
+ HasSHA bool
_ CacheLinePad
}
diff --git a/src/internal/cpu/cpu_x86.go b/src/internal/cpu/cpu_x86.go
index 1582e83..f6d4e06 100644
--- a/src/internal/cpu/cpu_x86.go
+++ b/src/internal/cpu/cpu_x86.go
@@ -37,6 +37,7 @@
cpuid_BMI2 = 1 << 8
cpuid_ERMS = 1 << 9
cpuid_ADX = 1 << 19
+ cpuid_SHA = 1 << 29
// edx bits for CPUID 0x80000001
cpuid_RDTSCP = 1 << 27
@@ -57,6 +58,7 @@
{Name: "pclmulqdq", Feature: &X86.HasPCLMULQDQ},
{Name: "popcnt", Feature: &X86.HasPOPCNT},
{Name: "rdtscp", Feature: &X86.HasRDTSCP},
+ {Name: "sha", Feature: &X86.HasSHA},
{Name: "sse3", Feature: &X86.HasSSE3},
{Name: "sse41", Feature: &X86.HasSSE41},
{Name: "sse42", Feature: &X86.HasSSE42},
@@ -112,6 +114,7 @@
X86.HasBMI2 = isSet(ebx7, cpuid_BMI2)
X86.HasERMS = isSet(ebx7, cpuid_ERMS)
X86.HasADX = isSet(ebx7, cpuid_ADX)
+ X86.HasSHA = isSet(ebx7, cpuid_SHA)
var maxExtendedInformation uint32
maxExtendedInformation, _, _, _ = cpuid(0x80000000, 0)
To view, visit change 353402. To unsubscribe, or for help writing mail filters, visit settings.
Martin Möhrmann (Gerrit)
Oct 5, 2021, 12:55:55 PM10/5/21
to Gerrit Bot, Dirkjan Bussink, goph...@pubsubhelper.golang.org, Filippo Valsorda, Keith Randall, Martin Möhrmann, Adam Langley, Katie Hockman, Roland Shoemaker, Go Bot, golang-co...@googlegroups.com
Attention is currently required from: Martin Möhrmann, Keith Randall, Filippo Valsorda.
3 comments:
Patchset:
only looked at internal/cpu usage.
File src/crypto/sha256/sha256block_amd64.s:
Patch Set #1, Line 1077:
PBLENDWThis is SSE4.1 and therefore needs to check the corresponding cpu feature too on entry to this assembler.
Patch Set #1, Line 1079:
PALIGNRThis is SSSE3 and therefore needs to check the corresponding cpu feature too on entry to this assembly (useSHA)
To view, visit change 353402. To unsubscribe, or for help writing mail filters, visit settings.
Gerrit Bot (Gerrit)
Oct 5, 2021, 1:50:04 PM10/5/21
to Dirkjan Bussink, goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention is currently required from: Martin Möhrmann, Keith Randall, Filippo Valsorda.
Gerrit Bot uploaded patch set #2 to this change.
GitHub-Last-Rev: 8a216b40bfa9f14f9c660a3b726022bdc407d03f
GitHub-Pull-Request: golang/go#48720
---
M src/crypto/sha256/sha256block_amd64.go
M src/crypto/sha1/sha1block_amd64.go
M src/crypto/sha256/sha256block_amd64.s
M src/internal/cpu/cpu_x86.go
M src/internal/cpu/cpu.go
M src/crypto/sha1/sha1block_amd64.s
M src/crypto/sha256/sha256block_decl.go
7 files changed, 604 insertions(+), 8 deletions(-)
To view, visit change 353402. To unsubscribe, or for help writing mail filters, visit settings.
Martin Möhrmann (Gerrit)
Oct 5, 2021, 1:57:56 PM10/5/21
to Gerrit Bot, Dirkjan Bussink, goph...@pubsubhelper.golang.org, Filippo Valsorda, Keith Randall, Martin Möhrmann, Adam Langley, Katie Hockman, Roland Shoemaker, Go Bot, golang-co...@googlegroups.com
Attention is currently required from: Dirkjan Bussink, Martin Möhrmann, Keith Randall, Filippo Valsorda.
1 comment:
File src/crypto/sha256/sha256block_amd64.s:
Patch Set #1, Line 1077:
PBLENDW
Ah ok, I wasn't sure about these kinds of checks. […]
It might be true now, it might not be true in the future. Its easier just to check. Its also possible for the user to request not to use any SSE4.1 instructions and it would be odd if we do not honor that.
To view, visit change 353402. To unsubscribe, or for help writing mail filters, visit settings.
Dirkjan Bussink (Gerrit)
Oct 5, 2021, 2:07:38 PM10/5/21
to Gerrit Bot, goph...@pubsubhelper.golang.org, Martin Möhrmann, Filippo Valsorda, Keith Randall, Martin Möhrmann, Adam Langley, Katie Hockman, Roland Shoemaker, Go Bot, golang-co...@googlegroups.com
Attention is currently required from: Martin Möhrmann, Keith Randall, Filippo Valsorda.
2 comments:
File src/crypto/sha256/sha256block_amd64.s:
Patch Set #1, Line 1077:
PBLENDW
This is SSE4. […]
Ah ok, I wasn't sure about these kinds of checks. Fwiw, there are no CPUs I could find that have SHA extensions and aren't also recent enough to have SSE4.1 as well.
I have pushed up a fix for this and the other case mentioned to check these flags as well.
Patch Set #1, Line 1079:
PALIGNR
This is SSSE3 and therefore needs to check the corresponding cpu feature too on entry to this assemb […]
Done
To view, visit change 353402. To unsubscribe, or for help writing mail filters, visit settings.
Gerrit Bot (Gerrit)
Oct 30, 2021, 11:23:28 AM10/30/21
to Dirkjan Bussink, goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention is currently required from: Dirkjan Bussink, Martin Möhrmann, Keith Randall, Filippo Valsorda.
Gerrit Bot uploaded patch set #3 to this change.
GitHub-Last-Rev: 3512fd60d1b2688f166f961aa4913a4407920cd6
GitHub-Pull-Request: golang/go#48720
---
M src/crypto/sha256/sha256block_amd64.go
M src/crypto/sha1/sha1block_amd64.go
M src/crypto/sha256/sha256block_amd64.s
M src/internal/cpu/cpu_x86.go
M src/internal/cpu/cpu.go
M src/crypto/sha1/sha1block_amd64.s
M src/crypto/sha256/sha256block_decl.go
7 files changed, 603 insertions(+), 7 deletions(-)
To view, visit change 353402. To unsubscribe, or for help writing mail filters, visit settings.
Dirkjan Bussink (Gerrit)
Oct 30, 2021, 11:39:59 AM10/30/21
to Gerrit Bot, goph...@pubsubhelper.golang.org, Martin Möhrmann, Filippo Valsorda, Keith Randall, Martin Möhrmann, Adam Langley, Katie Hockman, Roland Shoemaker, Go Bot, golang-co...@googlegroups.com
Attention is currently required from: Martin Möhrmann, Keith Randall, Filippo Valsorda.
1 comment:
Patchset:
I have rebased this on the latest master since there was a merge conflict.
To view, visit change 353402. To unsubscribe, or for help writing mail filters, visit settings.
Dirkjan Bussink (Gerrit)
Nov 6, 2021, 3:34:17 PM11/6/21
to Gerrit Bot, goph...@pubsubhelper.golang.org, Martin Möhrmann, Filippo Valsorda, Keith Randall, Martin Möhrmann, Adam Langley, Katie Hockman, Roland Shoemaker, Go Bot, golang-co...@googlegroups.com
Attention is currently required from: Martin Möhrmann, Keith Randall, Filippo Valsorda.
1 comment:
Patchset:
Is there any interest in this? I guess it might be too late now for 1.18, but otherwise for 1.19?
To view, visit change 353402. To unsubscribe, or for help writing mail filters, visit settings.
Ben Schwartz (Gerrit)
Dec 1, 2021, 5:33:51 PM12/1/21
to Gerrit Bot, Dirkjan Bussink, goph...@pubsubhelper.golang.org, Martin Möhrmann, Filippo Valsorda, Keith Randall, Martin Möhrmann, Adam Langley, Katie Hockman, Roland Shoemaker, Go Bot, golang-co...@googlegroups.com
Attention is currently required from: Martin Möhrmann, Keith Randall, Filippo Valsorda.
1 comment:
Patchset:
I'm not a Go maintainer, so I have no authority here.
I would like to see this move forward, but I note that this implementation here looks a lot more repetitive than the existing assembly versions. Could this be condensed using a pattern like the SHA256ROUND0/1(...) macros? That might make it more appealing and consistent with the Go Assembly Policy.
To view, visit change 353402. To unsubscribe, or for help writing mail filters, visit settings.
Dirkjan Bussink (Gerrit)
Dec 2, 2021, 7:41:00 AM12/2/21
to Gerrit Bot, goph...@pubsubhelper.golang.org, Ben Schwartz, Martin Möhrmann, Filippo Valsorda, Keith Randall, Martin Möhrmann, Adam Langley, Katie Hockman, Roland Shoemaker, Gopher Robot, golang-co...@googlegroups.com
Attention is currently required from: Ben Schwartz, Martin Möhrmann, Keith Randall, Filippo Valsorda.
1 comment:
Patchset:
I'm not a Go maintainer, so I have no authority here. […]
I have thought about that and also read the policy, but in the end I did opt for the current setup.
It's mainly because it directly follows the documentation at https://software.intel.com/content/www/us/en/develop/articles/intel-sha-extensions.html and makes it at least imho easier follow the algorithm based on that documentation.
If maintainers here disagree though, I can look at how to refactor it. But at least personally I found the current code easier to follow than the indirections in the AVX2 implementation (but that one would be a lot more complex and elaborate if it was written out so the trade off I think is different there).
To view, visit change 353402. To unsubscribe, or for help writing mail filters, visit settings.
Gopher Robot (Gerrit)
Apr 1, 2023, 2:06:41 AM4/1/23
to Gerrit Bot, Dirkjan Bussink, goph...@pubsubhelper.golang.org, Ben Schwartz, Martin Möhrmann, Filippo Valsorda, Keith Randall, Martin Möhrmann, Adam Langley, Katie Hockman, Roland Shoemaker, golang-co...@googlegroups.com
Gopher Robot abandoned this change.
Abandoned
GitHub PR golang/go#48720 has been closed.
To view, visit change 353402. To unsubscribe, or for help writing mail filters, visit settings.
Reply all
Reply to author
Forward
0 new messages