[net] http2: optimize buffer allocation in transport
Gerrit Bot (Gerrit)
Gerrit Bot has uploaded this change for review.
http2: optimize buffer allocation in transport
We have identified a high memory usage problem in our production service, which utilizes Traefik as a gRPC proxy. This service handles a substantial volume of gRPC bi-directional streaming requests that can persist for extended periods, spanning many days. Currently, there exists only a single global buffer pool in the http2 package. The allocated buffers, regardless of their sizes, are shared among requests with vastly different characteristics. For instance, gRPC streaming requests typically require smaller buffer sizes and occupy buffers for significant durations. Conversely, general HTTP requests may necessitate larger buffer sizes but only retain them temporarily. Unfortunately, the large buffers allocated by HTTP requests are can be chosen for subsequent gRPC streaming requests, resulting in numerous large buffers being unable to be recycled.
In our production environment, which processes approximately 1 million gRPC streaming requests, memory usage can soar to an excessive 800 GiB. This is a substantial waste of resources.
To address this challenge, we propose implementing a multi-layered buffer pool mechanism. This mechanism allows requests with varying characteristics to select buffers of appropriate sizes, optimizing resource allocation and recycling.
Change-Id: I834f7c08d90fd298aac7971ad45dc1a36251788b
GitHub-Last-Rev: 17d112893f3f1421950ae7532ad0c9b2562b38cd
GitHub-Pull-Request: golang/net#182
---
M http2/transport.go
1 file changed, 22 insertions(+), 4 deletions(-)
diff --git a/http2/transport.go b/http2/transport.go
index b20c749..5568144 100644
--- a/http2/transport.go
+++ b/http2/transport.go
@@ -19,6 +19,7 @@
"io/fs"
"log"
"math"
+ "math/bits"
mathrand "math/rand"
"net"
"net/http"
@@ -1680,7 +1681,23 @@
return int(n) // doesn't truncate; max is 512K
}
-var bufPool sync.Pool // of *[]byte
+// Seven buf pools for different frame sizes to avoid cases like long-running streaming requests using small frame size
+// occupy big buffer initially allocated for previous requests which do need big buffer.
+// The size ranges are as following:
+// {0 KB, 16 KB], {16 KB, 32 KB], {32 KB, 64 KB], {64 KB, 128 KB], {128 KB, 256 KB], {256 KB, 512 KB], {512 KB, infinity}
+var bufPools [7]sync.Pool // of *[]byte
+func bufPoolIndex(size int) int {
+ if size <= 16384 {
+ return 0
+ }
+ size -= 1
+ bits := bits.Len(uint(size))
+ index := bits - 14
+ if index >= len(bufPools) {
+ return len(bufPools) - 1
+ }
+ return index
+}
func (cs *clientStream) writeRequestBody(req *http.Request) (err error) {
cc := cs.cc
@@ -1698,12 +1715,13 @@
// Scratch buffer for reading into & writing from.
scratchLen := cs.frameScratchBufferLen(maxFrameSize)
var buf []byte
- if bp, ok := bufPool.Get().(*[]byte); ok && len(*bp) >= scratchLen {
- defer bufPool.Put(bp)
+ index := bufPoolIndex(scratchLen)
+ if bp, ok := bufPools[index].Get().(*[]byte); ok && len(*bp) >= scratchLen {
+ defer bufPools[index].Put(bp)
buf = *bp
} else {
buf = make([]byte, scratchLen)
- defer bufPool.Put(&buf)
+ defer bufPools[index].Put(&buf)
}
var sawEOF bool
To view, visit change 508415. To unsubscribe, or for help writing mail filters, visit settings.
Gopher Robot (Gerrit)
Congratulations on opening your first change. Thank you for your contribution!
Next steps:
A maintainer will review your change and provide feedback. See
https://go.dev/doc/contribute#review for more info and tips to get your
patch through code review.
Most changes in the Go project go through a few rounds of revision. This can be
surprising to people new to the project. The careful, iterative review process
is our way of helping mentor contributors and ensuring that their contributions
have a lasting impact.
During May-July and Nov-Jan the Go project is in a code freeze, during which
little code gets reviewed or merged. If a reviewer responds with a comment like
R=go1.11 or adds a tag like "wait-release", it means that this CL will be
reviewed as part of the next development cycle. See https://go.dev/s/release
for more details.
Gerrit Bot (Gerrit)
Attention is currently required from: Damien Neil, Tom Bergan.
Gerrit Bot uploaded patch set #2 to this change.
http2: optimize buffer allocation in transport
We have identified a high memory usage problem in our production service, which utilizes Traefik as a gRPC proxy. This service handles a substantial volume of gRPC bi-directional streaming requests that can persist for extended periods, spanning many days. Currently, there exists only a single global buffer pool in the http2 package. The allocated buffers, regardless of their sizes, are shared among requests with vastly different characteristics. For instance, gRPC streaming requests typically require smaller buffer sizes and occupy buffers for significant durations. Conversely, general HTTP requests may necessitate larger buffer sizes but only retain them temporarily. Unfortunately, the large buffers allocated by HTTP requests are can be chosen for subsequent gRPC streaming requests, resulting in numerous large buffers being unable to be recycled.
In our production environment, which processes approximately 1 million gRPC streaming requests, memory usage can soar to an excessive 800 GiB. This is a substantial waste of resources.
To address this challenge, we propose implementing a multi-layered buffer pool mechanism. This mechanism allows requests with varying characteristics to select buffers of appropriate sizes, optimizing resource allocation and recycling.
Change-Id: I834f7c08d90fd298aac7971ad45dc1a36251788b
GitHub-Last-Rev: 13def8859edea092665cecfdb3735b7af92f80cf
GitHub-Pull-Request: golang/net#182
---
M http2/transport.go
1 file changed, 22 insertions(+), 4 deletions(-)
To view, visit change 508415. To unsubscribe, or for help writing mail filters, visit settings.
Damien Neil (Gerrit)
Attention is currently required from: Tom Bergan.
Patch set 2:Run-TryBot +1Code-Review +2
1 comment:
Patchset:
Looks like a reasonable improvement to me.
We really should have a good, common pool for []byte so this doesn't need to be reinvented in every package, but the perfect doesn't need to be the enemy of the good.
To view, visit change 508415. To unsubscribe, or for help writing mail filters, visit settings.
Brad Fitzpatrick (Gerrit)
Attention is currently required from: Tom Bergan.
Patch set 2:Code-Review +1
1 comment:
File http2/transport.go:
Patch Set #2, Line 1716:
scratchLen := cs.frameScratchBufferLen(maxFrameSize)this returns at max 512K, so the infinity pool should never be used, right? Maybe leave a comment that the last pool should never be used in practice and is just for safety/paranoia if somebody cranks up the scratch buffer max size later?
To view, visit change 508415. To unsubscribe, or for help writing mail filters, visit settings.
Gerrit Bot (Gerrit)
Attention is currently required from: Brad Fitzpatrick, Damien Neil, Tom Bergan.
Gerrit Bot uploaded patch set #3 to this change.
The following approvals got outdated and were removed: Run-TryBot+1 by Damien Neil, TryBot-Result+1 by Gopher Robot
http2: optimize buffer allocation in transport
We have identified a high memory usage problem in our production service, which utilizes Traefik as a gRPC proxy. This service handles a substantial volume of gRPC bi-directional streaming requests that can persist for extended periods, spanning many days. Currently, there exists only a single global buffer pool in the http2 package. The allocated buffers, regardless of their sizes, are shared among requests with vastly different characteristics. For instance, gRPC streaming requests typically require smaller buffer sizes and occupy buffers for significant durations. Conversely, general HTTP requests may necessitate larger buffer sizes but only retain them temporarily. Unfortunately, the large buffers allocated by HTTP requests are can be chosen for subsequent gRPC streaming requests, resulting in numerous large buffers being unable to be recycled.
In our production environment, which processes approximately 1 million gRPC streaming requests, memory usage can soar to an excessive 800 GiB. This is a substantial waste of resources.
To address this challenge, we propose implementing a multi-layered buffer pool mechanism. This mechanism allows requests with varying characteristics to select buffers of appropriate sizes, optimizing resource allocation and recycling.
Change-Id: I834f7c08d90fd298aac7971ad45dc1a36251788b
GitHub-Last-Rev: 477197698f27f55a1cffe6864fcb84582f80c7a7
GitHub-Pull-Request: golang/net#182
---
M http2/transport.go
1 file changed, 26 insertions(+), 4 deletions(-)
To view, visit change 508415. To unsubscribe, or for help writing mail filters, visit settings.
傅承堯 (Gerrit)
Attention is currently required from: Brad Fitzpatrick, Damien Neil, Tom Bergan.
2 comments:
Patchset:
Thanks for your suggestion. I have added suggested comments to the `bufPoolIndex()` function.
File http2/transport.go:
Patch Set #2, Line 1716:
scratchLen := cs.frameScratchBufferLen(maxFrameSize)
this returns at max 512K, so the infinity pool should never be used, right? Maybe leave a comment th […]
Done. I have added suggested comments to the `bufPoolIndex()` function.
To view, visit change 508415. To unsubscribe, or for help writing mail filters, visit settings.
Damien Neil (Gerrit)
Attention is currently required from: Brad Fitzpatrick, Tom Bergan.
Patch set 3:Run-TryBot +1Code-Review +2
To view, visit change 508415. To unsubscribe, or for help writing mail filters, visit settings.
Damien Neil (Gerrit)
Attention is currently required from: Brad Fitzpatrick, Tom Bergan.
Patch set 4:Run-TryBot +1Auto-Submit +1Code-Review +2
Gopher Robot (Gerrit)
Gopher Robot submitted this change.
http2: optimize buffer allocation in transport
We have identified a high memory usage problem in our production service, which utilizes Traefik as a gRPC proxy. This service handles a substantial volume of gRPC bi-directional streaming requests that can persist for extended periods, spanning many days. Currently, there exists only a single global buffer pool in the http2 package. The allocated buffers, regardless of their sizes, are shared among requests with vastly different characteristics. For instance, gRPC streaming requests typically require smaller buffer sizes and occupy buffers for significant durations. Conversely, general HTTP requests may necessitate larger buffer sizes but only retain them temporarily. Unfortunately, the large buffers allocated by HTTP requests are can be chosen for subsequent gRPC streaming requests, resulting in numerous large buffers being unable to be recycled.
In our production environment, which processes approximately 1 million gRPC streaming requests, memory usage can soar to an excessive 800 GiB. This is a substantial waste of resources.
To address this challenge, we propose implementing a multi-layered buffer pool mechanism. This mechanism allows requests with varying characteristics to select buffers of appropriate sizes, optimizing resource allocation and recycling.
Change-Id: I834f7c08d90fd298aac7971ad45dc1a36251788b
GitHub-Last-Rev: 477197698f27f55a1cffe6864fcb84582f80c7a7
GitHub-Pull-Request: golang/net#182
Reviewed-on: https://go-review.googlesource.com/c/net/+/508415
Run-TryBot: Damien Neil <dn...@google.com>
Reviewed-by: David Chase <drc...@google.com>
Reviewed-by: Brad Fitzpatrick <brad...@golang.org>
TryBot-Result: Gopher Robot <go...@golang.org>
Auto-Submit: Damien Neil <dn...@google.com>
Reviewed-by: Damien Neil <dn...@google.com>
---
M http2/transport.go
1 file changed, 26 insertions(+), 4 deletions(-)
diff --git a/http2/transport.go b/http2/transport.go
index b20c749..b0d482f 100644
--- a/http2/transport.go
+++ b/http2/transport.go
@@ -19,6 +19,7 @@
"io/fs"
"log"
"math"
+ "math/bits"
mathrand "math/rand"
"net"
"net/http"
@@ -1680,7 +1681,27 @@
return int(n) // doesn't truncate; max is 512K
}
-var bufPool sync.Pool // of *[]byte
+// Seven bufPools manage different frame sizes. This helps to avoid scenarios where long-running
+// streaming requests using small frame sizes occupy large buffers initially allocated for prior
+// requests needing big buffers. The size ranges are as follows:
+// {0 KB, 16 KB], {16 KB, 32 KB], {32 KB, 64 KB], {64 KB, 128 KB], {128 KB, 256 KB],
+// {256 KB, 512 KB], {512 KB, infinity}
+// In practice, the maximum scratch buffer size should not exceed 512 KB due to
+// frameScratchBufferLen(maxFrameSize), thus the "infinity pool" should never be used.
+// It exists mainly as a safety measure, for potential future increases in max buffer size.
+var bufPools [7]sync.Pool // of *[]byte
+func bufPoolIndex(size int) int {
+ if size <= 16384 {
+ return 0
+ }
+ size -= 1
+ bits := bits.Len(uint(size))
+ index := bits - 14
+ if index >= len(bufPools) {
+ return len(bufPools) - 1
+ }
+ return index
+}
func (cs *clientStream) writeRequestBody(req *http.Request) (err error) {
cc := cs.cc
@@ -1698,12 +1719,13 @@
// Scratch buffer for reading into & writing from.
scratchLen := cs.frameScratchBufferLen(maxFrameSize)
var buf []byte
- if bp, ok := bufPool.Get().(*[]byte); ok && len(*bp) >= scratchLen {
- defer bufPool.Put(bp)
+ index := bufPoolIndex(scratchLen)
+ if bp, ok := bufPools[index].Get().(*[]byte); ok && len(*bp) >= scratchLen {
+ defer bufPools[index].Put(bp)
buf = *bp
} else {
buf = make([]byte, scratchLen)
- defer bufPool.Put(&buf)
+ defer bufPools[index].Put(&buf)
}
var sawEOF bool
To view, visit change 508415. To unsubscribe, or for help writing mail filters, visit settings.
David Chase (Gerrit)
Attention is currently required from: Brad Fitzpatrick, Tom Bergan.
Patch set 4:Code-Review +1